Kibana启动后报错 [circuit_breaking_exception] [parent] Data too large

报错详情如下：

[root@bogon bin]# log [08:16:50.962] [error][task_manager] Failed to poll for work: [circuit_breaking_exception] [parent] Data too large, data for [<http_request>] would be [1062289576/1013mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1062287472/1013mb], new bytes reserved: [2104/2kb], usages [request=0/0b, fielddata=276072/269.6kb, in_flight_requests=22366578/21.3mb, accounting=31202776/29.7mb], with { bytes_wanted=1062289576 & bytes_limit=1020054732 & durability="PERMANENT" } :: {"path":"/.kibana_task_manager/_update_by_query","query":{"ignore_unavailable":true,"refresh":true,"max_docs":10,"conflicts":"proceed"},"body":"{\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"term\":{\"task.status\":\"idle\"}},{\"range\":{\"task.runAt\":{\"lte\":\"now\"}}}]}},{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"term\":{\"task.status\":\"running\"}},{\"term\":{\"task.status\":\"claiming\"}}]}},{\"range\":{\"task.retryAt\":{\"lte\":\"now\"}}}]}}]}},{\"bool\":{\"should\":[{\"exists\":{\"field\":\"task.interval\"}},{\"bool\":{\"must\":[{\"term\":{\"task.taskType\":\"vis_telemetry\"}},{\"range\":{\"task.attempts\":{\"lt\":3}}}]}},{\"bool\":{\"must\":[{\"term\":{\"task.taskType\":\"lens_telemetry\"}},{\"range\":{\"task.attempts\":{\"lt\":3}}}]}}]}}]}}]}},\"sort\":{\"_script\":{\"type\":\"number\",\"order\":\"asc\",\"script\":{\"lang\":\"expression\",\"source\":\"doc['task.retryAt'].value || doc['task.runAt'].value\"}}},\"seq_no_primary_term\":true,\"script\":{\"source\":\"ctx._source.task.ownerId=params.ownerId; ctx._source.task.status=params.status; ctx._source.task.retryAt=params.retryAt;\",\"lang\":\"painless\",\"params\":{\"ownerId\":\"ed68f18f-eec5-4965-a0e1-a81440027af1\",\"retryAt\":\"2024-10-22T08:17:20.888Z\",\"status\":\"claiming\"}}}","statusCode":429,"response":"{\"error\":{\"root_cause\":[{\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [<http_request>] would be [1062289576/1013mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1062287472/1013mb], new bytes reserved: [2104/2kb], usages [request=0/0b, fielddata=276072/269.6kb, in_flight_requests=22366578/21.3mb, accounting=31202776/29.7mb]\",\"bytes_wanted\":1062289576,\"bytes_limit\":1020054732,\"durability\":\"PERMANENT\"}],\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [<http_request>] would be [1062289576/1013mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1062287472/1013mb], new bytes reserved: [2104/2kb], usages [request=0/0b, fielddata=276072/269.6kb, in_flight_requests=22366578/21.3mb, accounting=31202776/29.7mb]\",\"bytes_wanted\":1062289576,\"bytes_limit\":1020054732,\"durability\":\"PERMANENT\"},\"status\":429}"}

表面原因是在ES中大量内存被占用（比如开启logstash的多线程写入的时候，在未达到默认FullGC的门槛的时候，会直接将ES熔断器熔断，因为熔断器的total.limit小于FulGC默认的内存的75%），GC无法对heap进行垃圾回收，导致node内存用量超出limit限制。

每次查询ES会将查询结果缓存到内存里面，方便下一次查询索引时更快速，如果一直不清理缓存的数据就有可能内存溢出。

解决方法为：

在elasticsearch.yml添加配置

# 缓存回收大小，无默认值
# 有了这个设置，最久未使用（LRU）的 fielddata 会被回收为新数据腾出空间
# 控制fielddata允许内存大小，达到HEAP 20% 自动清理旧cache，默认是无限缓存下去，肯定会超内存然后内存溢出
indices.fielddata.cache.size: 20%
# -----------------------------------主要是添加上面的那一个配置-------------------------------
indices.breaker.total.use_real_memory: false
# fielddata 断路器默认设置堆的 60% 作为 fielddata 大小的上限。
indices.breaker.fielddata.limit: 40%
# request 断路器估算需要完成其他请求部分的结构大小，例如创建一个聚合桶，默认限制是堆内存的 40%。
indices.breaker.request.limit: 40%
# total 揉合 request 和 fielddata 断路器保证两者组合起来不会使用超过堆内存的 70%(默认值)。
indices.breaker.total.limit: 95%

如果是es的内存不满足，需在jvm.options文件中调大es的内存设置。