logstash.outputs.elasticsearch报错异常2018-03-12

logstash-5.5.1

[2018-03-12T20:52:11,073][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://10.0.89.***:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://10.0.89.***:9200/, :error_message=>"Elasticsearch Unreachable: [http://10.0.89.***:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-03-12T20:52:11,074][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://10.0.89.***:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2018-03-12T20:52:11,081][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://10.0.89.***:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://10.0.89.***:9200/, :error_message=>"Elasticsearch Unreachable: [http://10.0.89.***:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

elasticsearch-5.5.1

[2018-03-13T09:16:09,476][WARN ][o.e.m.j.JvmGcMonitorService] [jt00232] [gc][85764] overhead, spent [4.4s] collecting in the last [
5.3s]
[2018-03-13T10:27:51,129][INFO ][o.e.c.m.MetaDataDeleteIndexService] [jt00232] [logstash-linuxhistory-2018.03.10/3cNRqlb_QTOq7AoKGC
v8WA] deleting index
[2018-03-13T10:27:54,657][INFO ][o.e.c.m.MetaDataIndexStateService] [jt00232] closing indices [[[logstash-linuxsecure-2018.03.11/Jz
yfcSc8TMqzkqBUPQsiOw]]]
[2018-03-13T10:27:57,951][INFO ][o.e.c.m.MetaDataIndexStateService] [jt00232] closing indices [[[logstash-linuxmessages-2018.03.11/
2sQ0crQYRFayNkj3eX3kMg]]]
[2018-03-13T10:28:01,254][INFO ][o.e.c.m.MetaDataIndexStateService] [jt00232] closing indices [[[logstash-linuxhistory-2018.03.11/w
V4bECUtQSuU4SW4eztA2Q]]]

[2018-03-13T10:42:33,448][WARN ][o.e.c.a.s.ShardStateAction] [jt00232] [logstash-linuxmessages-2018.03.12][3] received shard failed
for shard id [[logstash-linuxmessages-2018.03.12][3]], allocation id [rIMAzFOnRwucMjPbUn1x_A], primary term [1], message [failed to pe
rform indices:data/write/bulk[s] on replica [logstash-linuxmessages-2018.03.12][3], node[kVQQNk0sS8OCzOH-cVBAGw], [R], s[STARTED], a[id
=rIMAzFOnRwucMjPbUn1x_A]], failure [NodeDisconnectedException[[jt00211][10.0.89.***:9300][indices:data/write/bulk[s][r]] disconnect
ed]]
org.elasticsearch.transport.NodeDisconnectedException: [jt00211][10.0.89.***:9300][indices:data/write/bulk[s][r]] disconnected
[2018-03-13T10:42:33,448][WARN ][o.e.c.a.s.ShardStateAction] [jt00232] [logstash-linuxmessages-2018.03.12][3] received shard failed
for shard id [[logstash-linuxmessages-2018.03.12][3]], allocation id [rIMAzFOnRwucMjPbUn1x_A], primary term [1], message [failed to pe
rform indices:data/write/bulk[s] on replica [logstash-linuxmessages-2018.03.12][3], node[kVQQNk0sS8OCzOH-cVBAGw], [R], s[STARTED], a[id
=rIMAzFOnRwucMjPbUn1x_A]], failure [NodeDisconnectedException[[jt00211][10.0.89.***:9300][indices:data/write/bulk[s][r]] disconnected]]

 

 

系统12号早上就出现异常了

[2018-03-12T09:24:29,308][WARN ][r.suppressed ] path: /elastalert_status/elastalert_status, params: {index=elastalert_status, type=elastalert_status}
org.elasticsearch.transport.NodeNotConnectedException: [jtelk***][10.0.89.***:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationActio
n.java:751) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performRemoteAction(TransportReplicatio
nAction.java:692) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:6
60) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.
java:816) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:311) ~[elastics
earch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:139) ~[elasticsearch-5.5.1.jar:5.
5.1]
at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:111) ~[elasticsearch-5.5.1.jar:5.
5.1]
at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:103) ~[elasticsearch-5.5.1.jar:5.
5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:8
02) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationA
ction.java:781) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067) ~[ela
sticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:893) ~[elasticsearch-
5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-
5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
[2018-03-12T09:24:29,485][WARN ][o.e.c.a.s.ShardStateAction] [jtelkvpra02] [logstash-ldapapacheaccess-2018.03.08][2] received shard fai
led for shard id [[logstash-ldapapacheaccess-2018.03.08][2]], allocation id [UrTiNLtXSO6PximRlQwZpw], primary term [1], message [failed
to perform indices:data/write/bulk[s] on replica [logstash-ldapapacheaccess-2018.03.08][2], node[kVQQNk0sS8OCzOH-cVBAGw], [R], s[START
ED], a[id=UrTiNLtXSO6PximRlQwZpw]], failure [NodeDisconnectedException[[jtelk***][10.0.89.***:9300][indices:data/write/bulk[s][r]] d
isconnected]]

 

 

kafka中堆积了很多数据，持续观察