Linux安装ELK-Elasticsearch

Elasticsearch安装部署

  Elasticsearch是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。要负责数据存储与搜索。
  最近在学习ELK方面的知识,在安装的过程中也遇到了很多坑,网上找了很多Elasticsearch的安装文档,但是或多或少都有些没有说清楚的地方,于是结合自己的部署过程,整理成自己的文档,以便后续查阅。

1、Elasticsearch安装所需环境

  Elasticsearch对于JAVA JDK环境有要求,需要JDK1.8或以上的支持。操作系统官网上都有各个系统的安装文件。我本机的测试环境是CentOS 7.3,JDK版本是1.8.0_131

1 [root@localhost /]# more /etc/redhat-release 
2 CentOS Linux release 7.3.1611 (Core) 
3 [root@localhost /]# java -version
4 java version "1.8.0_131"
5 Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
6 Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

     JDK建议使用Oracle的,不要使用CentOS自带的OpenJDK,如果使用java -version查看是OpenJDK的,可以先卸载,然后再安装Oracle JDK。

1 先查看 rpm -qa | grep java,如果openjdk,则可使用yum remove来删除
2 [root@localhost /]# rpm -qa | grep java
3 java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
4 [root@localhost /]#yum -y remove java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5

  删除完OpenJDK之后,则可以到Oracle官网下载JDK:http://www.oracle.com/technetwork/java/javase/downloads/index.html
  下载JDK之后,解压文件,并设置PATH、JAVA_HOME既可以配置完成。

 1 [root@localhost /]# tar -zxvf jdk-8u151-linux-x64.tar.gz
 2 将JDK路径加入环境变量中
 3 [root@localhost /]vim /etc/profile
 4 将下面内容复制到文件最后一行:
 5 export JAVA_HOME=/usr/local/jdk1.7.0_67      #这里换成你的JDK解压路径
 6 export PATH=$PATH:$JAVA_HOME/bin
 7 编辑完后,刷新配置文件
 8 [root@localhost /]source /etc/profile
 9 完成,查看JDK版本
10 [root@localhost /]java -version

2、Elasticsearch下载安装及配置

  • Elasticsearch下载解压

  从ELK官网下载Elasticsearch:https://www.elastic.co/cn/downloads/elasticsearch
  下载elasticsearch-6.1.0.tar.gz的tar包后,在Centos中解压

1 [root@localhost local]# tar -zxvf elasticsearch-6.1.0.tar.gz
2 [root@localhost elasticsearch-6.1.0]# pwd
3 /usr/local/elasticsearch-6.1.0
4 [root@localhost elasticsearch-6.1.0]# ls
5 bin  config  data  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.textile

  Elasticsearch的tar包是已经编译好的,下载后直接使用即可

  • Elasticsearch配置文件

  现在我们来配置 config/elasticsearch.yml文件,Elasticsearch的所有配置信息都在此文件中。

 1 [root@localhost config]# more elasticsearch.yml
 2 # ======================== Elasticsearch Configuration =========================
 3 #
 4 # NOTE: Elasticsearch comes with reasonable defaults for most settings.
 5 #       Before you set out to tweak and tune the configuration, make sure you
 6 #       understand what are you trying to accomplish and the consequences.
 7 #
 8 # The primary way of configuring a node is via this file. This template lists
 9 # the most important settings you may want to configure for a production cluster.
10 #
11 # Please consult the documentation for further information on configuration options:
12 # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
13 #
14 # ---------------------------------- Cluster -----------------------------------
15 #
16 # Use a descriptive name for your cluster:
17 #
18 cluster.name: jun-application
19 #
20 # ------------------------------------ Node ------------------------------------
21 #
22 # Use a descriptive name for the node:
23 #
24 node.name: node-1
25 #
26 # Add custom attributes to the node:
27 #
28 node.attr.rack: r1
29 #
30 # ----------------------------------- Paths ------------------------------------
31 #
32 # Path to directory where to store the data (separate multiple locations by comma):
33 #
34 #path.data: /path/to/data
35 #
36 # Path to log files:
37 #
38 #path.logs: /path/to/logs
39 #
40 # ----------------------------------- Memory -----------------------------------
41 #
42 # Lock the memory on startup:
43 #
44 #bootstrap.memory_lock: true
45 #
46 # Make sure that the heap size is set to about half the memory available
47 # on the system and that the owner of the process is allowed to use this
48 # limit.
49 #
50 # Elasticsearch performs poorly when the system is swapping the memory.
51 #
52 # ---------------------------------- Network -----------------------------------
53 #
54 # Set the bind address to a specific IP (IPv4 or IPv6):
55 #
56 network.host: 10.1.129.101
57 #
58 # Set a custom port for HTTP:
59 #
60 http.port: 9200
61 #
62 # For more information, consult the network module documentation.
63 #
64 # --------------------------------- Discovery ----------------------------------
65 #
66 # Pass an initial list of hosts to perform discovery when new node is started:
67 # The default list of hosts is ["127.0.0.1", "[::1]"]
68 #
69 #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
70 #
71 # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
72 #
73 #discovery.zen.minimum_master_nodes: 
74 #
75 # For more information, consult the zen discovery module documentation.
76 #
77 # ---------------------------------- Gateway -----------------------------------
78 #
79 # Block initial recovery after a full cluster restart until N nodes are started:
80 #
81 #gateway.recover_after_nodes: 3
82 #
83 # For more information, consult the gateway module documentation.
84 #
85 # ---------------------------------- Various -----------------------------------
86 #
87 # Require explicit names when deleting indices:
88 #
89 #action.destructive_requires_name: true
90 
91 http.cors.enabled: true
92 http.cors.allow-origin: "*"

  这里ES配置就结束了,对没有错,就这么简单。当然以上的配置只是基本的配置,还有更多的参数设置可以到官网上了解更多。

  • Elasticsearch创建普通用户

  配置完成后,即可以启动Elasticsearch,但是在启动之前需要先创建一个用户,并将此用户的权限赋予Elasticsearch的目录。(主要是因为Elasticsearch不能用root用户来启动,必须用非root用户)

1 [root@localhost /]# useradd elkuser
2 #elasticsearch 只能用非 root 启动
3 [root@localhost /]#  chown -R elkuser.elkuser elasticsearch-6.1.0

  用户创建后,进入到elasticsearch-6.1.0目录启动ES:

1 [root@localhost elasticsearch-6.1.0]# pwd
2 /usr/local/elasticsearch-6.1.0
3 [root@localhost elasticsearch-6.1.0]# cd bin
4 [root@localhost bin]# ls
5 elasticsearch      elasticsearch-env.bat       elasticsearch-plugin       elasticsearch-service-mgr.exe  elasticsearch-translog.bat
6 elasticsearch.bat  elasticsearch-keystore      elasticsearch-plugin.bat   elasticsearch-service-x64.exe
7 elasticsearch-env  elasticsearch-keystore.bat  elasticsearch-service.bat  elasticsearch-translog
8 [root@localhost bin]# ./elasticsearch
  • Elasticsearch文件打开数及堆大小检测

  在ES启动过程中可能会报如下的错误:

1 ERROR: [2] bootstrap checks failed
2 [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
3 [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

  错误主要说明,linux中elasticsearch最大文件打开数太小,需要我们修改到对应的数值:

 1 1.max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
 2 
 3 修改/etc/security/limits.conf文件,添加或修改如下行:
 4 *        hard    nofile           65536
 5 *        soft    nofile           65536
 6 
 7 2.max virtual memory areas vm.max……
 8 
 9 修改 /etc/sysctl.conf 文件,添加如下行:
10 
11 vm.max_map_count=262144
12 修改好了以后,运行/sbin/sysctl -p
13 
14 重启以后,再启动es即可,就可以通过主机ip访问。

  如果是内存不足就需要调整内存大小了或者调整config/jvm.options的参数

1 在elasticsearch-6.1.0/config目录中有jvm.options文件,可以设置JVM大小
2 [root@localhost config]# pwd
3 /usr/local/elasticsearch-6.1.0/config
4 [root@localhost config]# ls
5 elasticsearch.yml  jvm.options  log4j2.properties
6 [root@localhost config]# vi jvm.options
7 -Xms2g
8 -Xmx2g

  最大堆内存和最小堆内存两者值设定为一至,同时尽可能大,同时不要超过32G,最大堆内存和最小堆内存如果不一致,在启动中的时候会进行内存大小自动调整,可能会出现中断的情况,为了避免此情况的产生,所以heap_check中要求最大内存最小内存相当,本例中设置为2G。

  • 启动Elasticsearch
    前序工作全部准备完成后,即可在elasticsearch-6.1.0目录中的bin目录下执行elasticsearch-6.1.0文件:
1 [root@localhost elasticsearch-6.1.0]# pwd
2 /usr/local/elasticsearch-6.1.0
3 [root@localhost elasticsearch-6.1.0]# cd bin
4 [root@localhost bin]# ls
5 elasticsearch      elasticsearch-env.bat       elasticsearch-plugin       elasticsearch-service-mgr.exe  elasticsearch-translog.bat
6 elasticsearch.bat  elasticsearch-keystore      elasticsearch-plugin.bat   elasticsearch-service-x64.exe
7 elasticsearch-env  elasticsearch-keystore.bat  elasticsearch-service.bat  elasticsearch-translog
8 [root@localhost bin]# ./elasticsearch
9 [root@localhost bin]# ./elasticsearch -d 可以使其在后台运行

  浏览器访问 http://localhost:9200 ,可以查看到对应的节点信息,如下显示则说明启动正常:

3、Elasticsearch安装Head插件

  Elasticsearch-head是一个界面化的集群操作和管理工具,可以对集群进行傻瓜式操作。你可以通过插件把它集成到es(首选方式),也可以安装成一个独立webapp。
  ES-head主要有四个方面的操作:

  1. 显示集群的拓扑,并且能够执行索引和节点级别操作
  2. 搜索接口能够查询集群中原始json或表格格式的检索数据
  3. 能够快速访问并显示集群的状态
  4. 有一个输入窗口,允许任意调用RESTful API。这个接口包含几个选项,可以组合在一起以产生有趣的结果;
     - 请求方法(get、put、post、delete),查询json数据,节点和路径
     - 支持JSON验证器
     - 支持重复请求计时器
     - 支持使用javascript表达式变换结果
     - 收集结果的能力随着时间的推移(使用定时器),或比较的结果
     - 能力图表转换后的结果在一个简单的条形图(包括时间序列)

  Elasticsearch-head的官方文档:https://github.com/mobz/elasticsearch-head

安装Elasticsearch的Head插件,首先需要在Centos中安装Git、Nodejs、grunt。三个软件全部安装配置完成后,才可安装Head插件

1 $ tar -zxf git-1.7.2.2.tar.gz
2 $ cd git-1.7.2.2
3 $ make prefix=/usr/local all
4 $ sudo make prefix=/usr/local install
5 以上命令执行完成后,即可使用Git

在编译安装过程中,可能会出现如下错误:
Can't locate ExtUtils/MakeMaker.pm in @INC…………
解决方法如下:
yum install perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker

  • Nodejs安装
    从Nodejs官网中下载Nodejs安装包:http://nodejs.cn/download/
    Nodejs 官网提供了编译好的Linux二进制包,你也可以下载下来直接应用。下载二进制的包,直接解压到目录即可:
1 [root@localhost local]# tar -xvf node-v8.9.0-linux-x64.tar.xz
2 [root@localhost node-v8.9.0]# pwd
3 /usr/local/node-v8.9.0
4 [root@localhost node-v8.9.0]# ls
5 bin  CHANGELOG.md  etc  include  lib  LICENSE  README.md  share

配置NODE_HOME,进入profile编辑环境变量

1 vim /etc/profile

设置nodejs环境变量

1 #set for nodejs
2 export NODE_HOME=/usr/local/node-v8.9.0
3 export PATH=$NODE_HOME/bin:$PATH

:wq保存并退出,编译/etc/profile 使配置生效

1 source /etc/profile

验证是否安装配置成功

1 node -v

输出node-v8.9.0表示配置成功

  • Grunt安装
    安装还Nodejs后,直接在CentsOS中运行如下命令即可安装Grunt
1 npm install -g grunt-cli
2 grunt -version   -- 安装后 ,查看 grunt版本。

-g代表全局安装,并且自动加入PATH变量。安装完成后检查一下。
grunt是一个很方便的构建工具,可以进行打包压缩、测试、执行等等的工作,Elasticsearch里的head插件就是通过grunt启动的,因此需要安装grunt。

  • 下载 head 插件的源码并安装
1 git clone git://github.com/mobz/elasticsearch-head.git

下载之后会在目录中生成elasticsearch-head文件夹

1 [root@localhost elasticsearch-head]# pwd
2 /usr/local/elasticsearch-head
3 [root@localhost elasticsearch-head]# ls
4 Dockerfile         elasticsearch-head.sublime-project  grunt_fileSets.js  LICENCE       package.json                  proxy           _site  test
5 Dockerfile-alpine  Gruntfile.js                        index.html         node_modules  plugin-descriptor.properties  README.textile  src
6 [root@localhost elasticsearch-head]# 

下载之后,需要修改head源码。因为直接执行有很多限制,比如无法跨机器访问。因此需要用户修改两个地方。
elasticsearch-head/Gruntfile.js,增加hostname属性

 1 connect: {
 2     server: {
 3         options: {
 4             port: 9100,
 5             hostname: '*',
 6             base: '.',
 7             keepalive: true
 8         }
 9     }
10 }

elasticsearch-head/_site/app.js。修改head的连接地址

1 this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://localhost:9200";
2 把localhost修改成你es的服务器地址,如:
3 this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://10.1.129.101:9200";

然后在elasticsearch-head源码目录中,执行npm install:

1 npm install

在运行npm install时,可能会存在Head插件phantomjs权限问题:

 1 [root@localhost elasticsearch-head]# npm install
 2 phantomjs-prebuilt@2.1.16 install /usr/local/elasticsearch-head/node_modules/phantomjs-prebuilt
 3 node install.js
 4 PhantomJS not found on PATH
 5 Download already available at /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
 6 Verified checksum of previously downloaded file
 7 Extracting tar contents (via spawned process)
 8 Removing /usr/local/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom
 9 Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1513568757772/phantomjs-2.1.1-linux-x86_64 -> /usr/local/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom
10 Phantom installation failed { Error: EACCES: permission denied, link '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1513568757772/phantomjs-2.1.1-linux-x86_64' -> '/usr/local/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom'
11   errno: -13,
12   code: 'EACCES',
13   syscall: 'link',
14   path: '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1513568757772/phantomjs-2.1.1-linux-x86_64',
15   dest: '/usr/local/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom' } Error: EACCES: permission denied, link '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1513568757772/phantomjs-2.1.1-linux-x86_64' -> '/usr/local/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom'
16 npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression
17 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/fsevents):
18 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
19 
20 npm ERR! code ELIFECYCLE
21 npm ERR! errno 1
22 npm ERR! phantomjs-prebuilt@2.1.16 install: `node install.js`
23 npm ERR! Exit status 1
24 npm ERR! 
25 npm ERR! Failed at the phantomjs-prebuilt@2.1.16 install script.
26 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
27 
28 npm ERR! A complete log of this run can be found in:
29 npm ERR!     /root/.npm/_logs/2017-12-18T03_46_03_878Z-debug.log

解决方法,在npm install命令后加 -g 参数:

1 npm install -g

最后,在elasticsearch-head源代码目录下启动nodejs,运行 grunt server。
运行成功后,访问 http://localhost:9100 网站,即可看到elasticsearch的相关信息:

 

posted @ 2017-12-19 00:08  归来似少年  阅读(4763)  评论(1编辑  收藏  举报