使用Nginx+keepalived反向代理Tomcat实现负载均衡
1. 测试概要
本次测试使用4台虚拟机进行Nginx+Keepalived实现Tomcat的负载均衡与反向代理；
1.1. 测试环境
操作系统：CentOS 7 Minimal Install（Linux version 3.10.0-1160.el7.x86_64 ）
虚拟机VM1：Tomcat1（172.17.0.234）apache-tomcat-10.0.6 jdk-8u202-linux-x64.tar.gz
虚拟机VM2：Tomcat2（172.17.0.235）apache-tomcat-10.0.6 jdk-8u202-linux-x64.tar.gz
虚拟机VM3：Nginx+Keepalived(Master)（172.17.0.236）
虚拟机VM4：Nginx+Keepalived(Backup)（172.17.0.237）
VIP：172.17.0.238
1.2 系统设置（每台虚拟机都要执行以下操作）
（1）关闭防火墙
# systemctl stop firewalld ## 关闭firewalld服务，当前生效，重启失效
# systemctl disable firewalld ## 禁止firewalld开机启动
（2）禁用Selinux
# getenforce ## 获取selinux的状态
# setenforce 0 ## 临时禁用selinux
# vi /etc/selinux/config ## 修改selinux配置，禁止开机启动，修改SELINUX=disabled
（文件内容如下）
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
（3）配置IP地址，保证测试机器间的网络是通的
# vi /etc/sysconfig/network-scripts/ifcfg-ens33
网卡配置文件如下（根据实际情况配置IP地址）：
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=172.17.0.234
NETMASK=255.255.255.0
GATEWAY=172.17.0.254
DNS1=223.5.5.5
# service network restart ## 重启网络使配置生效
# ping 172.17.0.235 ## VM之间相互ping一下确认通讯正常
PING 172.17.0.235 (172.17.0.235) 56(84) bytes of data.
64 bytes from 172.17.0.235: icmp_seq=1 ttl=64 time=0.732 ms
64 bytes from 172.17.0.235: icmp_seq=2 ttl=64 time=0.536 ms
64 bytes from 172.17.0.235: icmp_seq=3 ttl=64 time=0.819 ms
1.3 JDK安装(VM1,VM2做同样配置)
# tar -zxvf jdk-8u202-linux-x64.tar.gz ## 解压jdk包
# mv jdk1.8.0_202/ /usr/local/ ## 将jdk移到/usr/local目录下
# vi /etc/profile ## 修改java的环境变量
以下为文件部分内容，在文件末尾追加（JAVA_HOME,CLASSPATH,PATH）环境变量）：
HOSTNAME=`/usr/bin/hostname 2>/dev/null`
HISTSIZE=1000
if [ "$HISTCONTROL" = "ignorespace" ] ; then
export HISTCONTROL=ignoreboth
else
export HISTCONTROL=ignoredups
fi
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL
# By default, we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then
umask 002
else
umask 022
fi
for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
if [ -r "$i" ]; then
if [ "${-#*i}" != "$-" ]; then
. "$i"
else
. "$i" >/dev/null
fi
fi
done
unset i
unset -f pathmunge
export JAVA_HOME=/usr/local/jdk1.8.0_202
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
1.4 Apache安装(VM1,VM2做同样配置)
# cd /usr/local/
# tar -zxvf apache-tomcat-10.0.6.tar.gz
# cd /usr/local/apache-tomcat-10.0.6/bin
# ./startup.sh
# vi /usr/local/apache-tomcat-10.0.6/webapps/ROOT/index.jsp
编辑index.jsp文件(VM1)
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>成功</title>
<h1>我是172.17.0.234（Tomcat-Master）
编辑index.jsp文件(VM2)
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>成功</title>
</h1><h1>我是172.17.0.235（Tomcat-Backup）
1.5 通过访问VM1，VM2的地址，确认Tomcat工作正常
★至此，VM1，VM2配置完成，接下来需要配置Nginx反向代理，测试环境注意一定要关闭firewalld和selinux，不然会带来很多麻烦，当然，生产环境建议配置防火墙策略来打通主机间的网络，而不是粗暴的关闭防火墙。
2. Nginx和keepalived安装部署（VM3，VM4同样配置）
##通过yum安装nginx和keepalived
# yum install keepalived
# yum install nginx
# vi /etc/nginx/nginx.conf
文件内容如下：
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
sendfile on;
keepalive_timeout 65;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
upstream www.test.com{
server 172.17.0.234:8080 weight=5;
server 172.17.0.235:8080 weight=5;
}
server {
listen 80;
server_name www.test.com:8080;
location / {
proxy_pass http://www.test.com;
}
error_page 404 /404.html;
location = /404.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
# systemctl start nginx ##启动nginx
# systemctl status nginx ##查看Nginx启动状态
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2021-06-16 22:22:49 EDT; 6h ago
Process: 1527 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Process: 1524 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 1522 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 1529 (nginx)
Tasks: 3
CGroup: /system.slice/nginx.service
├─1529 nginx: master process /usr/sbin/nginx
├─1530 nginx: worker process
└─1531 nginx: worker process
Jun 16 22:22:49 localhost.localdomain systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jun 16 22:22:49 localhost.localdomain nginx[1524]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Jun 16 22:22:49 localhost.localdomain nginx[1524]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Jun 16 22:22:49 localhost.localdomain systemd[1]: Started The nginx HTTP and reverse proxy server.
VM3,VM4反向代理到Tomcat，目前配置轮询权重是相同的server 172.17.0.234:8080 weight=5; server 172.17.0.235:8080 weight=5;
VM3,VM4反向代理到Tomcat，目前配置轮询权重是不相同的 server 172.17.0.234:8080 weight=2; server 172.17.0.235:8080 weight=8;
配置Keepalived服务
VM3配置：
# vi /etc/keepalived/keepalived.conf
文件内容如下：
! Configuration File for keepalived
global_defs {
router_id nginxmaster
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 62
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.17.0.238
}
}
# systemctl start keepalived ##启用keepalived
# systemctl status keepalived ##查看keepalived启动情况
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-06-17 08:25:46 EDT; 20s ago
Process: 7687 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 7688 (keepalived)
Tasks: 3
CGroup: /system.slice/keepalived.service
├─7688 /usr/sbin/keepalived -D
├─7689 /usr/sbin/keepalived -D
└─7690 /usr/sbin/keepalived -D
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
VM4配置：
# vi /etc/keepalived/keepalived.conf
文件内容如下：
! Configuration File for keepalived
global_defs {
router_id nginxbackup
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 62
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.17.0.238
}
}
# systemctl start keepalived ##启用keepalived
# systemctl status keepalived ##查看keepalived启动情况
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-06-17 08:30:20 EDT; 6s ago
Process: 2870 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 2871 (keepalived)
Tasks: 3
CGroup: /system.slice/keepalived.service
├─2871 /usr/sbin/keepalived -D
├─2872 /usr/sbin/keepalived -D
└─2873 /usr/sbin/keepalived -D
Jun 17 08:30:20 localhost.localdomain Keepalived_healthcheckers[2872]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 17 08:30:20 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) Entering MASTER STATE
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) setting protocol VIPs.
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
通过模拟VM1，VM3关机，系统正常工作，恢复后，系统负载正常；
浙公网安备 33010602011771号