ElasticSearch

ElasticSearch部署服务与配置

项目中遇到了需要大数据量的查询需求，原本使用的PostgreSQL在查询性能上不符合要求，于是就使用了ElasticSearch

ElasticSearch的底层是使用java开发的，所以需要安装java的JDK

 

安装java
yum install java-1.8.0-openjdk*

配置环境变量
vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el6_10.x86_64
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

使配置文件生效：source /etc/profile

验证java版本
java -version

 

 

 

开始是将ElasticSearch通过drf-haystack集成到django项目中，过程中发现这个库因为elasticsearch在2.x以后的版本中弃用了一个变量，在5.x后的版本删除了这个变量，2.x以后的版本会报警告，5.x以后的版本会报错误，导致只能完美支持1.x的版本，所以决定将ElasticSearch独立成单独的服务，前端使用dsl语言直接访问服务进行查询，后台使用elasticsearch的库进行插入数据，这里实在CentOS6.9下使用的6.7.1版本，大家可以选用适合自己的版本，下载地址：https://www.elastic.co/cn/downloads/elasticsearch

1.ElasticSearch不可以用root用户来启动，这里创建了一个用户

groupadd esuser
useradd esuser -g esuser -p esuser12345

修改密码 Lantucx2018
passwd esuser

2.将下载下来的压缩包拷贝到esuser的目录下，更改文件的权限，使esuser可以有权限操作Elasticsearch

tar -xzvf elasticsearch-6.7.1.tar.gz

chown -R esuser:esuser elasticsearch-6.7.1

3.修改配置文件

vim  elasticsearch-6.7.1/config/elasticsearch.yml
ip改为 192.168.1.110
将端口号改为 8200

注意要在Memory下面: 
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false

#开启跨域访问支持，默认为false
http.cors.enabled: true
http.cors.allow-origin: "*"

node.max_local_storage_nodes: 100

#或跨域访问允许的域名地址，(允许所有域名)以上使用正则
http.cors.allow-origin: /.*/ 

修改heap_size:
vim elasticsearch-6.7.1/config/jvm.options
修改允许使用的内存
-Xms32g
-Xmx32g

4.切换用户，开启ElasticSearch

su esuser
cd elasticsearch-6.7.1/bin
./elasticsearch

下面是我遇到的一些错误：

如果报错如下:
[2019-04-18T19:32:06,924][INFO ][o.e.t.TransportService   ] [YWePpHM] publish_address {192.168.164.176:9300}, bound_addresses {192.168.164.176:9300}
[2019-04-18T19:32:06,964][INFO ][o.e.b.BootstrapChecks    ] [YWePpHM] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max number of threads [1024] for user [elsearch] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2019-04-18T19:32:07,039][INFO ][o.e.n.Node               ] [YWePpHM] stopping ...
[2019-04-18T19:32:07,110][INFO ][o.e.n.Node               ] [YWePpHM] stopped
[2019-04-18T19:32:07,110][INFO ][o.e.n.Node               ] [YWePpHM] closing ...
[2019-04-18T19:32:07,153][INFO ][o.e.n.Node               ] [YWePpHM] closed
[2019-04-18T19:32:07,159][INFO ][o.e.x.m.p.NativeController] [YWePpHM] Native controller process has stopped - no new native processes can be started


解决：
5.1、文件句柄不足
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
su root 
cp /etc/security/limits.conf /etc/security/limits.conf.bak
vi /etc/security/limits.conf
备注：使用最高权限 修改安全配置 在文件末尾加入
# End of file
esuser       hard        nofile        65536 
esuser       soft        nofile        65536
*            soft        nproc         4096
*            hard        nproc         4096 

备注： esuser为用户名 可以是使用*进行通配  
nofile 最大打开文件数目

5.2、启动最大线程数限制
[2]: max number of threads [3802] for user [esuser] is too low, increase to at least [4096]
切换到root用户下
cd /etc/security/limits.d/
cp 90-nproc.conf 90-nproc.conf.bak

修改90-nproc.conf（系统不同文件名有变化）：
vim /etc/security/limits.d/90-nproc.conf
修改 * soft nproc 1024为
* soft nproc 4096

5.4.设定虚存不足
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
step1:修改安全限制配置文件
cp /etc/sysctl.conf /etc/sysctl.conf.bak
vim /etc/sysctl.conf

备注：行末加上vm.max_map_count = 262144
保存退出
使配置生效：sysctl -p

如下：
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

vm.max_map_count = 262144
备注：vm.max_map_count = 262144 值大于错误提示值65530

5.5、
ERROR: [1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Centos6不支持SecComp，而ES5.2.0后的版本默认bootstrap.system_call_filter为true，在elasticsearch.yml中增加bootstrap.system_call_filter为false，注意要在Memory下面:
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false

5.6、启动
su esuser
/elasticsearch-6.3.0/bin/elasticsearch -d -p pid

服务搭建好以后这里使用python的elasticsearch库导入的数据，推荐一个谷歌插件，用来查看Es服务，查询索引状态，构建查询条件都比较方便，叫elasticsearch-head