HAProxy+keepalived+nginx实现高性能负载均衡集群 转发

原文链接：https://mp.weixin.qq.com/s/33jhtkzAo9NMAlWY1gyjPw

1、环境规划

192.168.52.15	master	haproxy+keepalived
192.168.52.16	backup	haproxy+keepalived
192.168.52.17	web1	nginx1
192.168.52.18	web2	nginx2
192.168.52.88		vip

2、nginx安装

2.1、下载二进制包

wget -O /usr/local/src/nginx-1.24.0.tar.gz https://nginx.org/download/nginx-1.24.0.tar.gz

2.2、安装依赖包

yum install -y vim lrzsz tree screen psmisc lsof tcpdump wget ntpdate gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools iotop bc zip unzip zlib-devel bash-completion nfs-utils automake libxml2 libxml2-devel libxslt libxslt-devel perl perl-ExtUtils-Embed

2.3、安装nginx服务器

cd /usr/local/src

tar zxvf nginx-1.24.0.tar.gz

cd nginx-1.24.0/

./configure --prefix=/usr/local/nginx \

  --user=nginx \

  --group=nginx \

  --with-http_ssl_module \

  --with-http_v2_module \

  --with-http_realip_module \

  --with-http_stub_status_module \

  --with-http_gzip_static_module \

  --with-pcre \

  --with-stream \

  --with-stream_ssl_module \

  --with-stream_realip_module

make && make install

2.4、添加Nginx用户

useradd nginx -s /sbin/nologin -u 2000

chown -R nginx:nginx /usr/local/nginx/

2.5、配置Nginx服务

cat > /usr/lib/systemd/system/nginx.service <<EOF[Unit]Description=The nginx HTTP and reverse proxy serverAfter=network.target remote-fs.target nss-lookup.target [Service]Type=forkingPIDFile=/var/run/nginx.pidExecStartPre=/usr/bin/rm -f /var/run/nginx.pidExecStartPre=/usr/local/nginx/sbin/nginx -tExecStart=/usr/local/nginx/sbin/nginxExecReload=/bin/kill -s HUP KillSignal=SIGQUITTimeoutStopSec=5KillMode=processPrivateTmp=true [Install]WantedBy=multi-user.targetEOF

创建nginx命令软链接：

ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx

2.6、修改Nginx配置文件

vim /usr/local/nginx/conf/nginx.confuser  nginx;worker_processes  auto; error_log  /usr/local/nginx/logs/error.log warn;pid        /var/run/nginx.pid;  events {    worker_connections  1024;}  http {    include      /usr/local/nginx/conf/mime.types;    default_type  application/octet-stream;    sendfile        on;    server_tokens off;    tcp_nopush  on;    tcp_nodelay on;    # nginx日志格式    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                      '$status $body_bytes_sent "$http_referer" '                      '"$http_user_agent" "$http_x_forwarded_for"';     access_log  /usr/local/nginx/logs/access.log  main;     #sendfile        on;    #tcp_nopush     on;     keepalive_timeout  1d;    proxy_set_header   Host             $host;    proxy_set_header   X-Real-IP        $remote_addr;    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;    proxy_set_header  x-client-ip  $remote_addr;     #gzip  on;     include /usr/local/nginx/conf/conf.d/*.conf;}

mkdir /usr/local/nginx/conf/conf.d

vi /usr/local/nginx/conf/conf.d/default.conf

server {

     listen       80;

     server_name  localhost;

     access_log  logs/access.log  main;

     location / {

       root   /var/www/html;

       index  index.html index.htm;

    }

}

2.7、配置前端

mkdir /var/www/html -p

Nginx01配置:

echo "welcome to nginx01" > /var/www/html/index.html

Nginx02配置:

echo "welcome to nginx02" > /var/www/html/index.html

2.8、启动Nginx服务

systemctl daemon-reload

systemctl start nginx

systemctl status nginx

systemctl enable nginx

2.9、验证nginx

ps -ef |grep nginx

journalctl -u nginx

2.10、防火墙设置

firewall-cmd --permanent --add-port=80/tcp

firewall-cmd --permanent --add-port=443/tcp

firewall-cmd --reload

firewall-cmd --list-all-zones

2.11、访问nginx

3、harproxy+keepalived

3.1、开启路由转发功能

vim /etc/sysctl.conf

net.ipv4.ip_forward = 1

net.ipv4.conf.all.send_redirects = 0

net.ipv4.conf.default.send_redirects = 0

net.ipv4.conf.ens33.send_redirects = 0

sysctl -p

3.2、安装HAProxy和Keepalived

yum install -y haproxy keepalived

3.3、配置HAProxy

vi /etc/haproxy/haproxy.cfgglobal    log /dev/log local0    log /dev/log local1 notice    chroot /var/lib/haproxy    user haproxy    group haproxy    daemon defaults    log     global    mode    http    option  httplog    option  dontlognull    timeout connect 5000    timeout client  50000    timeout server  50000 frontend http_front    bind *:80    default_backend http_back backend http_back    balance roundrobin    server web1 192.168.52.17:80 check    server web2 192.168.52.18:80 check

3.4、配置keepalived

主节点（192.168.52.15）:

vi /etc/keepalived/keepalived.conf! Configuration File for keepalived global_defs {     router_id SERVER1} vrrp_instance VI_1 {      state MASTER      interface ens33      virtual_router_id 51      priority 100      advert_int 1      authentication {          auth_type PASS          auth_pass 1234      }      virtual_ipaddress {          192.168.52.88      }}

备节点（192.168.52.16）:

vi /etc/keepalived/keepalived.conf! Configuration File for keepalived global_defs {     router_id SERVER2} vrrp_instance VI_1 {      state BACKUP      interface ens33      virtual_router_id 51      priority 90      advert_int 1      authentication {          auth_type PASS          auth_pass 1234      }      virtual_ipaddress {          192.168.52.88      }}

3.5、启动服务

systemctl enable haproxy

systemctl start haproxy

systemctl status haproxy

systemctl enable keepalived

systemctl start keepalived

systemctl status keepalived

3.6、防火墙配置

systemctl disable firewalld

systemctl stop firewalld

3.7、测试负载均衡

keepalived主节点（192.168.52.15）：成功获取vip:192.168.52.88

keepalived备节点（192.168.52.16）：没有获取vip,正常

浏览器访问：http://192.168.52.88

3.8、防火墙放行配置

harproxy两台服务器防火墙配置：

firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 \

  --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

  

firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 \

  --out-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

 

firewall-cmd --zone=public --add-port=80/tcp --permanent

firewall-cmd --reload

 

nginx两台服务器防火墙配置：

firewall-cmd --zone=public --add-port=80/tcp --permanent

firewall-cmd --reload

 

查看防火墙配置：

iptables -L OUTPUT_direct --line-numbers

iptables -L INPUT_direct --line-numbers

删除防火墙配置：

firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 \

  --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

  

firewall-cmd --direct --permanent --remove-rule ipv4 filter OUTPUT 0 \

  --out-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

firewall-cmd --zone=public --remove-port=80/tcp --permanent

firewall-cmd --reload