logstash数据无法写入到es

现象：偶发性某天应用服务日志数据丢失，查看es无数据

架构：filebeat--->logstash--->es

logstash 上报错现象：

2023-10-13T13:05:14,161][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"srgp-2023.10.13", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x3b1e2406>], :response=>{"index"=>{"_index"=>"srgp-2023.10.13", "_type"=>"_doc", "_id"=>"X41sJ4sBPtPNo0ugp5is", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [logdate] of type [date] in document with id 'X41sJ4sBPtPNo0ugp5is'. Preview of field's value: '2023-10-13 13:05:13,009'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2023-10-13 13:05:13,009] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}
[2023-10-13T13:05:14,161][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"srgp-2023.10.13", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x529a2802>], :response=>{"index"=>{"_index"=>"srgp-2023.10.13", "_type"=>"_doc", "_id"=>"9MlsJ4sBGw4njSrKpxWs", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [logdate] of type [date] in document with id '9MlsJ4sBGw4njSrKpxWs'. Preview of field's value: '2023-10-13 13:05:13,007'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2023-10-13 13:05:13,007] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}

 

应用日志：
2023-10-13 09:51:14,710 INFO [AYQqkUTKRJufCZpBDujhww] [00534e185d57476498936]xxxxxxxxx
2023-10-13 09:51:14,750 INFO [AYQqkUTKRJufCZpBDujhww] xxxxxx

咨询ChatGpt：

GPT的解决方法也不行，其实上面的配置我的logstash也有，还是报这个错。

出处： ## https://www.cnblogs.com/liucx/

解决方法：
经过排查，logstash获取日志时，会以日志的第一条记录作为es自动创建的映射字段，logdate的类型变为时间类型了，导致无法写入到es，后修改这个索引的模板解决，强制mapping中的logdate设置为text类型

查询索引模板
GET _template/srgp

修改索引模板（上面的问题，把正常的maaping拷贝进去）
PUT _template/srgp

数据丢失咋办：将丢失的日志sed出来，然后重新写到文件中，即可恢复

 解决哈