MySQL Router遭遇Too many open files错误引起监听端口异常案例.md
2026-09-29 11:55 潇湘隐者 阅读(11) 评论(0) 收藏 举报这个是几天前遇到的一个案例,MySQL Router的6446端口突然连接不上. 检查的时候,发现telnet 6446端口不通,而且MySQL Router服务正常,但是端口6446不见了.当时的现象如下所示:
# netstat -ntlp | grep mysqlrouter
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6447 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6448 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6449 0.0.0.0:* LISTEN 3409998/mysqlrouter
正常情况下,应该如下所示:
# netstat -ntlp | grep mysqlrouter
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6446 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6447 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6448 0.0.0.0:* LISTEN 3409998/mysqlrouter
tcp 0 0 0.0.0.0:6449 0.0.0.0:* LISTEN 3409998/mysqlrouter
检查告警邮件与MySQL Router日志发现有"generic:24, Too many open files"这类告警, 具体的错误日志如下所示:
...............................................................
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing INFO [7f78587e8700] Stop accepting connections for routing routing:bootstrap_x_rw listening on 0.0.0.0:6448
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing INFO [7f78587e8700] Stop accepting connections for routing routing:bootstrap_rw_split listening on 0.0.0.0:6450
2026-09-24 03:02:22 routing WARNING [7f7859feb700] No available servers found for PRIMARY routing
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] No available servers found for PRIMARY routing
2026-09-24 03:02:22 routing INFO [7f78587e8700] Stop accepting connections for routing routing:bootstrap_x_ro listening on 0.0.0.0:6449
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] No available servers found for PRIMARY routing
2026-09-24 03:02:22 routing INFO [7f78587e8700] Stop accepting connections for routing routing:bootstrap_ro listening on 0.0.0.0:6447
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing INFO [7f78587e8700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2026-09-24 03:02:22 routing WARNING [7f7859feb700] No available servers found for PRIMARY routing
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:22 metadata_cache INFO [7f78587e8700] Removing unused GR notification session to 'mysqldb01:33060'
2026-09-24 03:02:22 metadata_cache INFO [7f78587e8700] Removing unused GR notification session to 'mysqldb02:33060'
2026-09-24 03:02:22 metadata_cache INFO [7f78587e8700] Removing unused GR notification session to 'mysqldb03:33060'
2026-09-24 03:02:22 metadata_cache INFO [7f78587e8700] ... cleared current routing table as a precaution
2026-09-24 03:02:22 routing ERROR [7f786a115700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:54414 failed: no destinations: no more destinations
2026-09-24 03:02:22 routing ERROR [7f7869113700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:54388 failed: no destinations: no more destinations
2026-09-24 03:02:22 routing ERROR [7f7869914700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:54390 failed: no destinations: no more destinations
2026-09-24 03:02:27 metadata_cache WARNING [7f78587e8700] Error parsing stats_updates_frequency from the router_options: options.stats_updates_frequency='null'; not an unsigned int. Using default value
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] Connected with metadata server running on mysqldb01:7306
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] Potential changes detected in cluster after metadata refresh (view_id=1)
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] Target cluster(s) are part of a ClusterSet: accepting RW connections
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] Cluster '****': role of a cluster within a ClusterSet is 'primary';
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] Metadata for cluster '****' has 3 member(s), single-primary:
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] mysqldb01:7306 / 33060 - mode=RW
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] mysqldb02:7306 / 33060 - mode=RO
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] mysqldb03:7306 / 33060 - mode=RO
2026-09-24 03:02:27 routing INFO [7f78587e8700] Start accepting connections for routing routing:bootstrap_x_rw listening on '0.0.0.0:6448'
2026-09-24 03:02:27 routing INFO [7f78587e8700] Start accepting connections for routing routing:bootstrap_rw_split listening on '0.0.0.0:6450'
2026-09-24 03:02:27 routing INFO [7f78587e8700] Start accepting connections for routing routing:bootstrap_x_ro listening on '0.0.0.0:6449'
2026-09-24 03:02:27 routing INFO [7f78587e8700] Start accepting connections for routing routing:bootstrap_ro listening on '0.0.0.0:6447'
2026-09-24 03:02:27 routing INFO [7f78587e8700] Start accepting connections for routing routing:bootstrap_rw listening on '0.0.0.0:6446'
2026-09-24 03:02:27 metadata_cache INFO [7f78587e8700] Enabling GR notices for cluster '****' changes on node mysqldb01:33060
2026-09-24 03:02:27 metadata_cache WARNING [7f78587e8700] Failed connecting GR Notices listener on mysqldb02:33060; (err_code=2005; err_msg='No such host is known 'mysqldb02'')
2026-09-24 03:02:27 metadata_cache WARNING [7f78587e8700] Failed connecting GR Notices listener on mysqldb03:33060; (err_code=2005; err_msg='No such host is known 'mysqldb03'')
2026-09-24 03:02:27 routing ERROR [7f78637fe700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:53030 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785c7f0700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52502 failed: resolve(mysqldb01) failed after 446ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing INFO [7f785c7f0700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2026-09-24 03:02:27 routing ERROR [7f786b138700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52332 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing INFO [7f786b138700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2026-09-24 03:02:27 routing ERROR [7f785cff1700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52704 failed: resolve(mysqldb01) failed after 447ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785e7f4700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52906 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785d7f2700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52690 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f78627fc700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52584 failed: resolve(mysqldb01) failed after 446ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f7863fff700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:53016 failed: resolve(mysqldb01) failed after 447ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing INFO [7f78637fe700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2026-09-24 03:02:27 routing ERROR [7f7860ff9700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52616 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785dff3700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52674 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f786a916700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52974 failed: resolve(mysqldb01) failed after 443ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785fff7700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52632 failed: resolve(mysqldb01) failed after 453ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785f7f6700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52882 failed: connect(/* mysqldb01:7306 */): is quarantined, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f7861ffb700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52850 failed: connect(/* mysqldb01:7306 */): is quarantined, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f7862ffd700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:35392 failed: resolve(mysqldb01) failed after 445ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785eff5700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52896 failed: resolve(mysqldb01) failed after 444ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f78617fa700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52608 failed: resolve(mysqldb01) failed after 450ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f78688f1700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:53004 failed: resolve(mysqldb01) failed after 442ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785b7ee700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52750 failed: resolve(mysqldb01) failed after 444ms: Device or resource busy, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f785f7f6700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52462 failed: connect(10.***.*.116:7306 /* mysqldb01 */) failed after 10000ms: Connection timed out, end of destinations: no more destinations
2026-09-24 03:02:27 routing ERROR [7f786a916700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:54422 failed: no destinations: no more destinations
2026-09-24 03:02:28 routing INFO [7f7859feb700] Start accepting connections for routing routing:bootstrap_rw listening on '0.0.0.0:6446'
2026-09-24 03:02:29 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:29 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:29 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:29 routing INFO [7f785f7f6700] Stop accepting connections for routing routing:bootstrap_rw listening on 0.0.0.0:6446
2026-09-24 03:02:29 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:29 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:29 routing WARNING [7f7859feb700] accepting new connection failed at accept(): generic:24, Too many open files
2026-09-24 03:02:30 routing WARNING [7f7859feb700] [routing:bootstrap_rw] failed to open socket for 0.0.0.0:6446: Too many open files
2026-09-24 03:02:37 routing ERROR [7f7860ff9700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:53072 failed: connect(/* mysqldb01:7306 */): is quarantined, end of destinations: no more destinations
2026-09-24 03:02:37 routing ERROR [7f78617fa700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:53064 failed: connect(/* mysqldb01:7306 */): is quarantined, end of destinations: no more destinations
2026-09-24 03:02:37 routing ERROR [7f785fff7700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:53086 failed: connect(/* mysqldb01:7306 */): is quarantined, end of destinations: no more destinations
2026-09-24 03:02:37 routing ERROR [7f78627fc700] [routing:bootstrap_rw] connecting to backend(s) for client from 10.***.*.115:52838 failed: connect(/* mysqldb01:7306 */): is quarantined, end of destinations: no more destinations
从上面错误信息中可以看到,由于文件句柄耗尽,MySQL Router的rw 端口反复 Stop/Start；最终连监听socket都开不出来. 如下所示:
2026-09-24 03:02:30 routing WARNING [7f7859feb700] [routing:bootstrap_rw] failed to open socket for 0.0.0.0:6446: Too many open files
可能短时间内由于连接风暴.导致MySQL Router进程的句柄文件被耗尽,超过系统限制的文件数量(或网络连接、管道等).
# PID=$(pgrep -f mysqlrouter | head -1)
# cat /proc/$PID/limits | grep -i 'open files'
Max open files 1024 262144 files
# cat /proc/$PID/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 62588 62588 processes
Max open files 1024 262144 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 62588 62588 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
这种情况下,其实没有什么好的方法,只能先临时重启MySQL Router服务,重启后正常. 后续还需优化, 例如修改limits.conf,以及systemd服务中关于Max open files的配置.以防在特殊/极端情况下遇到Too many open files.
扫描上面二维码关注我
如果你真心觉得文章写得不错，而且对你有所帮助，那就不妨帮忙“推荐"一下，您的“推荐”和”打赏“将是我最大的写作动力！
本文版权归作者所有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接.
浙公网安备 33010602011771号