wrk 压测中请求无法响应问题解决过程

================= 遇到问题 =================
$ 直连压测

wrk -c10000 -t100 -d100m http://localhost:9981/order/list

$ nginx代理压测

wrk -c10000 -t100 -d100m http://localhost:8864/order/list

压测外的请求无法响应

================= 搜素方案 =================
$ 修改linux参数配置（最大打开文件数与进程最多打开文件数）以支持大量连接
$ nginx压测时会产生大量time_wait，与后端通信时开启keep alive以缓解time_wait情况

================= 用到的工具 =================
// 查看linux tcp连接情况

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}';

// 更快查看linux tcp连接情况的方式

ss -ant | awk 'NR>1 {++s[$1]} END {for(k in s) print k,s[k]}';

// 每秒输出一次当前系统连接状态

ss.sh
#!/bin/bash
while :
do
    ss -ant | awk 'NR>1 {++s[$1]} END {for(k in s) print k,s[k]}'
    sleep 1s;
done

================= 最终解决 =================
$ 修改linux参数配置

modprobe ip_conntrack

vi /etc/sysctl.conf
fs.file-max = 1024000
net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000
/sbin/sysctl -p

vi /etc/security/limits.conf
*         hard    nofile      1024000
*         soft    nofile      1024000

try
    net.netfilter.nf_conntrack_max = 1024000
    net.nf_conntrack_max = 1024000
instead
    net.ipv4.ip_conntrack_max = 1024000
    net.ipv4.netfilter.ip_conntrack_max = 1024000

$ 修改nginx配置

/etc/nginx/nginx.conf
user  nginx;
worker_processes  4;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections 10240;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/default.conf
upstream http_backend {
    server    127.0.0.1:9981;
    keepalive 512;
}

server {
    listen       8864;
    server_name  localhost;

    location /nginx {
        default_type   application/json;
        return 200     '{"success": true}';
    }

    location / {
        proxy_pass         http://http_backend;
        proxy_http_version 1.1;
        proxy_set_header   Connection "";
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

nginx -s reload

$ 再次压测，直连和nginx代理都能正常响应（虽然很慢...应该与硬件条件有关）

 

参考文章：

构建C1000K的服务器(1) – 基础

构建C1000K的服务器(2) – 实现百万连接的comet服务器

sysctl: cannot stat /proc/sys/net/ipv4/netfilter/ip_conntrack_max: No such file or directory