Fork me on GitHub

Elasticsearch 节点磁盘使用率过高,导致ES集群索引无副本


最近在查看线上的 es,发现最近2天的索引没有副本,集群的状态也是为 yellow 的。


es 所在的服务器磁盘是还有剩余空间的。只不过磁盘使用了大概 89%,按道理来说应该是会继续使用的,并创建索引的副本的,我们经过查阅官方文档


Controls the low watermark for disk usage. It defaults to 85%, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices or, specifically, any shards that have never previously been allocated.


Controls the high watermark. It defaults to 90%, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not.


Controls the flood stage watermark. It defaults to 95%, meaning that Elasticsearch enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node that has at least one disk exceeding the flood stage. This is a last resort to prevent nodes from running out of disk space. The index block must be released manually once there is enough disk space available to allow indexing operations to continue.

我们可以知道,es 集群的默认配置是当集群中的某个节点磁盘达到使用率为 85% 的时候,就不会在该节点进行创建副本,当磁盘使用率达到 90% 的时候,尝试将该节点的副本重分配到其他节点。当磁盘使用率达到95% 的时候,当前节点的所有索引将被设置为只读索引。


1. 扩大磁盘


2. 删除部分历史索引

可以看看 elasticsearch定时删除索引第二版

3. 更改es设置

  1. 更改配置文件(需要重启es)
  2. 动态更改(api,无需重启)

es 的设置默认是 85% 和 90 %,我们更改为 90%和 95%。


elasticsearch.yml 文件配置:

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 90%
cluster.routing.allocation.disk.watermark.high: 95%
cluster.routing.allocation.disk.watermark.flood_stage: 98%


所谓的动态更改就是通过 es 的 api 进行更改。transient 临时更改,persistent是永久更改。

api 接口 /_cluster/settings

注意 cluster.routing.allocation.disk.watermark.flood_stage 参数是 6.0 版本开始才有的,在5的版本是没有该配置的, 是不支持的,我在修改5.6 的版本的时候添加了该参数,是有错误返回的 "reason":"persistent setting [cluster.routing.allocation.disk.watermark.flood_stage], not dynamically updateable"},"status":4001. 5.6 版本官方文档链接:

查看es 当前的配置

查看es 当前的配置 get 请求 /_cluster/settings

	"persistent": {
		"xpack": {
			"monitoring": {
				"collection": {
					"enabled": "true"
	"transient": {
		"cluster": {
			"routing": {
				"allocation": {
					"disk": {
						"watermark": {
							"low": "90%",
							"high": "95%"
			"info": {
				"update": {
					"interval": "1m"

永久更改 persistent


    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "": "1m"

临时更改 transient


    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "": "1m"


root@111:~# curl -H "Content-Type: application/json"  -XPUT  -d '{"transient": {    "cluster.routing.allocation.disk.watermark.low": "90%", "cluster.routing.allocation.disk.watermark.high": "95%", "": "1m"}}'




posted @ 2020-01-04 22:07  自由早晚乱余生  阅读(2317)  评论(0编辑  收藏