Elasticsearch常见错误

错误1: Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]];

1 2 3 4 5 6 7 8 9 10 11 12 13

原因：elasticsearch.yml文件配置错误导致   解决：参数与参数值(等号)间需要空格 [root@elk-node03 ~]# vim /etc/elasticsearch/elasticsearch.yml ............... #node.name:elk-node03.kevin.cn         #错误 node.name: elk-node03.kevin.cn           #正确   #或者如下配置 #node.name ="elk-node03.kevin.cn"    #错误 #node.name = "elk-node03.kevin.cn"   #正确   然后重启elasticsearch服务

错误2: org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root

1 2 3 4 5 6 7 8 9 10 11 12

原因：处于对root用户的安全保护，需要使用其他用户组进行授权启动   解决： 用户组进行授权启动 [root@elk-node03 ~]# groupadd elasticsearch           [root@elk-node03 ~]# useradd elasticsearch -g elasticsearch -p elasticsearch [root@elk-node03 ~]# chown -R elasticsearch.elasticsearch /data/es-data                 #给es的数据目录授权, 否则es服务启动报错 [root@elk-node03 ~]# chown -R elasticsearch.elasticsearch/var/log/elasticsearch     #给es的日志目录授权, 否则es服务启动报错   以上是yum安装elasticsearch情况, 需要给elasticsearch的数据目录和日志目录授权, 如果elasticsearch是编译安装, 则需要给它的安装目录也授权   接着重启elasticsearch服务即可

错误3: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot a ...'(errno=12);

1 2 3 4 5 6 7 8 9 10 11 12 13 14

原因：jvm要分配最大内存超出系统内存   解决：适当调整指定jvm内存, 编辑elasticsearch 的jvm配置文件 # vim /data/elasticsearch/config/jvm.options -Xms8g -Xmx8g   如果是yum安装的elasticsearch, 则修改如下配置文件 [root@elk-node03 ~]# vim /etc/sysconfig/elasticsearch # Heap size defaults to 256m min, 1g max             #最小为1g # Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g     #设置为物理内存的50%, 但不要操作31g ES_HEAP_SIZE=8g   然后重启elasticsearch服务即可

错误4: ERROR: [3] bootstrap checks failed

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

详细报错: [INFO ][o.e.b.BootstrapChecks    ] [SUcoFrg] bound or publishing to a non-loopback address, enforcing bootstrap checks ERROR: [3] bootstrap checks failed [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2]: max number of threads [3802] for user [elsearch] is too low, increase to at least [4096] [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [2019-02-20T02:35:47,170][INFO ][o.e.n.Node               ] [SUcoFrg] stopping ... [2019-02-20T02:35:47,316][INFO ][o.e.n.Node               ] [SUcoFrg] stopped [2019-02-20T02:35:47,316][INFO ][o.e.n.Node               ] [SUcoFrg] closing ... [2019-02-20T02:35:47,336][INFO ][o.e.n.Node               ] [SUcoFrg] closed   原因：虚拟机限制用户的执行内存   解决： [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2]: max number of threads [3802] for user [elasticsearch] is too low, increase to at least [4096]   修改安全限制配置文件 (使用root最高权限 修改安全配置 在文件末尾加入) [root@elk-node03 ~]# vim /etc/security/limits.conf elasticsearch       hard        nofile        65536 elasticsearch       soft        nofile        65536 *               soft       nproc         4096 *               hard      nproc          4096   修改系统配置文件 [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]   [root@elk-node03 ~]# /etc/sysctl.conf        #注意下面的参数值大于错误提示值 vm.max_map_count = 655360   然后重启elasticsearch服务即可

错误5: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/elasticsearch-6.3.0/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

详细报错： [2019-02-20T04:23:25,003][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/elasticsearch-6.3.0/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0]   at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/elasticsearch-6.3.0/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:243) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.node.Node.<init>(Node.java:270) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]   ... 6 more   原因：线程占用 解决：重新启动 [root@elk-node03 ~]# ps -ef|grep elasticsearch|awk -F" " '{print $2}'|xargs kill -9 [root@elk-node03 ~]# systemctl start elasticsearch [root@elk-node03 ~]# systemctl restart elasticsearch   [root@elk-node03 ~]# ps -ef|grep -v grep|grep elasticsearch elastic+ 15586     1 11 Apr08 ?        03:06:12 /bin/java -Xms8g -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.6.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch   一定要确保elasticsearch是用非root账号启动的

错误6: [Godfrey Calthrop] All shards failed for phase: [query] [jr-2018.08.06][[jr-2018.08.06][2]] NoShardAvailableActionException[null]

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

详细报错: [2019-02-06 18:27:24,553][DEBUG][action.search            ] [Godfrey Calthrop] All shards failed for phase: [query] [jr-2018.08.06][[jr-2018.08.06][2]] NoShardAvailableActionException[null]     at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:129)     at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:115)     at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:47)     at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)     at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)     at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)     at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)     at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)     at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)     at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:582)     at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:85)     at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)     at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)     at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166)     at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)     at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)     at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:449)     at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)   问题解决 通过以上排查大概知道是历史索引数据处于 open 状态过多，从而导致ES的CPU，内存占用过高导致的不可用。   关闭不需要的索引，减少内存占用 [root@elk-node03 ~]# curl -XPOST "http://10.0.8.44:9200/index_name/_close"   如果发现在关闭非热点索引数据后，elasticSearch集群的健康值依然是"red"状态，这时候要想到: 可能索引的"red"状态可能会影响ES的状态. 接着查看elasticSearch索引健康状态, 发现果不其然 [root@elk-node03 ~]# curl GET http://10.0.8.44:9200/_cluster/health?level=indices   {     "cluster_name": "kevin-elk",     "status": "red",     "timed_out": false,     "number_of_nodes": 3,     "number_of_data_nodes": 3,     "active_primary_shards": 660,     "active_shards": 660,     "relocating_shards": 0,     "initializing_shards": 0,     "unassigned_shards": 9,     "delayed_unassigned_shards": 0,     "number_of_pending_tasks": 0,     "number_of_in_flight_fetch": 0,     "task_max_waiting_in_queue_millis": 0,     "active_shards_percent_as_number": 98.65470852017937,     "indices": {         "jr-2019.02.06": {             "status": "red",             "number_of_shards": 3,             "number_of_replicas": 0,             "active_primary_shards": 0,             "active_shards": 0,             "relocating_shards": 0,             "initializing_shards": 0,             "unassigned_shards": 3         }     } }   解决方法，删除上面命令中查看的有问题的那条索引数据（这条数据是排查问题期间产生的脏数据，索引直接删除） [root@elk-node03 ~]# curl -XDELETE 'http://10.0.8.44:9200/jr-2019.02.06'   注意: 应注意elasticSearch的索引状态以及服务器的监控，及时清理或者关闭不必要的索引数据，避免这种情况发生。