task jbd2/sdb-8:19693 blocked for more than 120 seconds

点击查看代码

Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task jbd2/sdb-8:19693 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task NIOServerCxn.Fa:2704 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task kafka-scheduler:2736 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task data-plane-kafk:2757 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task data-plane-kafk:2759 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task data-plane-kafk:2761 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task data-plane-kafk:2764 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task ReplicaFetcherT:2888 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task ReplicaFetcherT:4925 blocked for more than 120 seconds.
Apr 11 12:50:41 yewu-log-es-wldata03 kernel: INFO: task executor-Heartb:4930 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task jbd2/sdb-8:17908 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task VM Thread:20023 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20276 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20296 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20648 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20649 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20650 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20651 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20652 blocked for more than 120 seconds.
Apr 11 19:47:04 yewu-log-es-wldata03 kernel: INFO: task elasticsearch[y:20653 blocked for more than 120 seconds

由于日志写入bulk报错量过大，造成系统内存脏数据量激增。默认脏数据的使用量为%40 对于一个120Gb+ 内存主机来说，要将这么大量的数据落盘的话就会出现这样的报错。

1. 查看具体的块设备对应的调度策略并进行优化
   看了很多介绍noon 方式适合ssd 适合逻辑层已经进行过io优化的应用进行使用
   echo noop > /sys/dev/block/8:16/device/block/sdb/queue/scheduler
2. 更改内存缓存的使用率
   vm.dirty_background_ratio = 5
   vm.dirty_ratio = 10
3. 关闭等待时长防止io hung 造成的defunct
   kernel.hung_task_timeout_secs=0