K8S篇之三 kubeadm安装2master+1node集群-之2master节点上配置keepalived+nginx实现K8S-apiserver高可用
五、控制节点上配置keepalived+nginx实现api的高可用
根据环境规划，把keepalived+nginx安装在2个master节点上
5.1 安装nginx主备
在nflmaster1(192.168.10.201)、nflmaster2(192.168.10.202)上作nginx的主备。
[root@nflmaster1 ~]# yum -y install keepalived nginx nginx-all-modules.noarch #nflmaster1节点安装keepalived、nginx服务、nginx的modulers服务（避免后面启动nginx报stream错误）
[root@nflmaster2 ~]# yum -y install keepalived nginx nginx-all-modules.noarch #nflmaster2节点安装
若nginx包没有安装成功，看一下/etc/yum.repos.d/目录下有没有epel.repo这个文件。没有的话，在/root/repo.bak/目录下把epel.repo文件拷贝到/etc/yum.repos.d/目录下。
5.2 修改主、备上的nginx配置文件，nginx配置都一样。
修改配置文件前，可以先备份
[root@nflmaster1 ~]# cp -rafp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak #先备份一份文件
[root@nflmaster1 ~]# vim /etc/nginx/nginx.conf #打开nginx.conf文件
打开后删除里面的内容，按小d，然后大G。--全部删除。
输入下面内容到文件中
点击查看nginx.conf代码
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.10.201:6443; # Master1 APISERVER IP:PORT
server 192.168.10.202:6443; # Master2 APISERVER IP:PORT
}
server {
listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
[root@nflmaster2 ~]# cd /etc/nginx/ #nflmaster2节点上操作
[root@nflmaster2 nginx]# mv nginx.conf nginx.conf.bak #备份下文件
[root@nflmaster2 nginx]# vim nginx.conf #新建nginx.conf文件，输入下面内容
点击查看nginx.conf代码
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.10.201:6443; # Master1 APISERVER IP:PORT
server 192.168.10.202:6443; # Master2 APISERVER IP:PORT
}
server {
listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
5.3 在主、备上配置keepalived
5.3.1 在主-nflmaster1节点上配置
[root@nflmaster1 ~]# cd /etc/keepalived/ #进入目录
[root@nflmaster1 keepalived]# mv keepalived.conf keepalived.conf.bak #先备份
[root@nflmaster1 keepalived]# vim keepalived.conf #对该目录下的keepalived.conf修改
点击查看主节点keepalived.conf代码
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.10.200/24
}
track_script {
check_nginx
}
}
5.3.2 在备-nflmster2节点上配置
[root@nflmaster2 ~]# cd /etc/keepalived/
[root@nflmaster2 keepalived]# mv keepalived.conf keepalived.conf.bak #备份好文件
[root@nflmaster2 keepalived]# vim keepalived.conf #新建keepalived.conf文件，输入以下内容
点击查看keepalived.conf代码
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.10.200/24
}
track_script {
check_nginx
}
}
5.4 根据上面keepalived.conf文件，创建check_nginx脚本文件
5.4.1 在主-nflmaster1上操作
[root@nflmaster1 keepalived]# vim /etc/keepalived/check_nginx.sh #在节点1上创建check_nginx.sh脚本
点击查看check_nginx.sh代码
#!/bin/bash
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
[root@nflmaster1 keepalived]# chmod +x /etc/keepalived/check_nginx.sh #在节点1上给权限
[root@nflmaster1 keepalived]# systemctl daemon-reload
[root@nflmaster1 keepalived]# systemctl start nginx #开启时报错
[root@nflmaster1 keepalived]# nginx -t
[root@nflmaster1 nginx]# yum -y install nginx-all-modules.noarch
[root@nflmaster1 nginx]# nginx -t #此时再次测试是OK的
[root@nflmaster1 nginx]# systemctl start nginx #master1节点上开启nginx
[root@nflmaster1 keepalived]# ss -antulp | grep 16443 #验证端口是否打开
[root@nflmaster1 keepalived]# systemctl enable nginx.service #设置节点1的nginx服务开机自启动
5.4.2 在备-nflmaster2上操作
[root@nflmaster2 ~]# cd /etc/keepalived/ #进入节点2上的keepalived配置文件目录
[root@nflmaster2 keepalived]# cat keepalived.conf #查看配置文件
[root@nflmaster2 keepalived]# vim /etc/keepalived/check_nginx.sh #新建同名的脚本文件check_nginx.sh，把以下内容输入
点击查看check_nginx.sh代码
#!/bin/bash
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
[root@nflmaster2 keepalived]# chmod +x /etc/keepalived/check_nginx.sh #加上执行权限
[root@nflmaster2 keepalived]# yum -y install nginx-all-modules.noarch #在这里先提前安装modules模块，上面如果已安装，则这一步省略
[root@nflmaster2 keepalived]# systemctl start nginx.service #开启nginx服务成功
[root@nflmaster2 keepalived]# systemctl status nginx.service #查看开启状态
[root@nflmaster2 keepalived]# ss -antulp | grep 16443
[root@nflmaster2 keepalived]# systemctl enable nginx.service #节点2上开机自启动
5.4.3 开启keepalived服务
①在节点1-nflmaster1上操作
[root@nflmaster1 keepalived]# systemctl start keepalived.service #节点1上，开启keepalived服务
[root@nflmaster1 keepalived]# systemctl status keepalived #查看节点1上keepalived状态
[root@nflmaster1 keepalived]# systemctl enable keepalived.service #节点1上设置开机自启动
[root@nflmaster1 keepalived]# ip -4 addr #查看网卡上vip设置成功
②在节点2-nflmaster2上操作
[root@nflmaster2 keepalived]# systemctl start keepalived
[root@nflmaster2 keepalived]# systemctl enable keepalived
[root@nflmaster2 keepalived]# systemctl status keepalived
[root@nflmaster2 keepalived]# ip -4 addr
备注：VIP只会在1台机器上，可以模拟master1节点故障，让漂移到节点2上
[root@nflmaster1 keepalived]# systemctl stop nginx #节点1上停止nginx服务
[root@nflmaster1 keepalived]# ip -4 addr #节点1上的网卡上没有vip
[root@nflmaster2 keepalived]# ip -4 addr | grep ens #节点2上查看网卡，可以看到VIP
[root@nflmaster1 keepalived]# systemctl start nginx #节点1上开启nginx服务
[root@nflmaster1 keepalived]# systemctl start keepalived #节点1上开启keealived服务
[root@nflmaster1 keepalived]# ip -4 addr | grep ens #VIP又漂移回来，因为keepalived的配置文件，master1的优先级高
5.4.4 扩展，解释check_nginx.sh脚本内容
[root@nflmaster1 keepalived]# cat /etc/keepalived/check_nginx.sh #在节点1上打开脚本
[root@nflmaster1 keepalived]# ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" #输出结果是1
[root@nflmaster1 keepalived]# ps -ef |grep nginx #查看节点1上nginx进程
[root@nflmaster1 keepalived]# ps -ef |grep nginx | grep sbin #把nginx进程有关的、再过滤下/usr/sbin/nginx，这个是nginx主进程，证明nginx成功。结果可以输出看到
[root@nflmaster1 keepalived]# ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" #有关nginx主进程有几条，再过滤下，输出结果是1
K8S篇之二 kubeadm安装2master+1node集群-之安装docker服务+安装初始化K8S需要的软件包
K8S篇之四 kubeadm安装2master+1node集群-之kubeadm安装控制节点+扩容控制节点+扩容工作节点+安装网络插件calico+安装coredns
浙公网安备 33010602011771号