Rancher-rke方式部署文档

Rancher-2.5.4  RKE方式部署

##########LB负载均衡软件包
local集群k8s版本kubectl-1.18
kubectl-1.18+ （执行kubectl查看local集群和配置等）
nginx-1.57
rke-1.2.5
docker-ce-18.09.9-3.el7.x86_64.rpm（docker 20版本存在问题）
helm-v3.5.0-linux-amd64.tar.gz


##########host文件
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.197.181  mirror.k8s.com
172.31.197.183 rancher.rke.com
172.31.197.184 rke184
172.31.197.185 rke185
172.31.197.186 rke186


1.配置hosts规划主机

2.执行init初始化服务器脚本（切换内核），安装ntp（local集群3台）

3.RKE方式部署rancher 需要新建普通用户 apps用户来执行rke up（rke config/rke remove）
apps用户需要免密到local集群（ssh-copyid操作）

4.rke config配置好后生成cluster.yml 如执行rke up 出错，根据错误信息，修改镜像版本等
终止TLS需要配置cluster.yml/ingress字段后 rke up
ingress:
  provider: nginx
  options:
    use-forwarded-headers: "true"




5.rke up 成功后保存生成的配置文件！ （cluster.rkestate / cluster.yml / kube_config_cluster.yml）

6.kubectl-1.18.4 copy /usr/local/bin/  新建apps目录和文件 .kube/config （copy 生成的kube_config_cluster.yml ->> /home/apps/.kube/config）

7.创建自签名证书使用官方生成脚本ca.sh （rancher页面证书有效期10年）https://docs.rancher.cn/docs/rancher2/installation_new/resources/advanced/self-signed-ssl/_index/#1-%E4%BB%80%E4%B9%88%E6%98%AF-https
./ca.sh --ssl-domain=rancher.rke.com  --ssl-size=2048 --ssl-date=3650
cacerts.pem  cakey.pem  openssl.cnf          rancher.rke.com.csr  tls.crt
cacerts.srl  ca.sh      rancher.rke.com.crt  rancher.rke.com.key  tls.key

8.配置nginx
yum -y install gcc zlib zlib-devel pcre-devel openssl openssl-devel
tar -zxvf nginx-1.15.7.tar.gz
cd nginx-1.15.7/
./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_gzip_static_module --with-stream --with-http_v2_module
make && make install

开机启动配置
vim  /lib/systemd/system/nginx.service

[Unit]
Description=nginx service
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s quit
PrivateTmp=true
[Install]
WantedBy=multi-user.target



nginx.conf配置：

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

http {
    upstream rancher {
        server 172.31.197.186:80;       ###后端local-k8s地址
        server 172.31.197.185:80;
        server 172.31.197.184:80;
    }

    map $http_upgrade $connection_upgrade {
        default Upgrade;
        ''      close;
    }

    server {
        listen 443 ssl http2;
        server_name rancher.rke.com;    ###rancher域名
        ssl_certificate /home/apps/ssl/ca/rancher.rke.com.crt;     ###证书路径
        ssl_certificate_key /home/apps/ssl/ca/rancher.rke.com.key; ###证书路径

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Port $server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://rancher;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
            proxy_read_timeout 900s;
            proxy_buffering off;
        }
    }

    server {
        listen 80;
        server_name rancher.rke.com;                       ###rancher域名
        return 301 https://$server_name$request_uri;
    }
}


启动nginx
systemctl start nginx
systemctl enable nginx



9.创建TLS-ca相关
kubectl create ns cattle-system
kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem


10.配置rancher模板参数
helm template rancher ./rancher-2.5.4.tgz --output-dir .     --namespace cattle-system     --set hostname=rancher.rke.com     --set rancherImage=mirror.k8s.com:8888/rancher/rancher     --set ingress.tls.source=secret     --set privateCA=true     --set systemDefaultRegistry=mirror.k8s.com:8888  --set useBundledSystemChart=true

11.启动rancher
kubectl -n cattle-system apply -R -f ./rancher

12.web访问即可，开始创建集群操作。


##########添加新集群
步骤：
设置主机名，配置hosts，切换内核init，新建apps用户，安装docker，配置/etc/docker/certs.d/mirror.k8s.com:8888/镜像库ca证书，配置ntp。
copy预安装kubectl版本1.19->/usr/local/bin/
执行rancher页面提供的命令即可。
出现cluster-agent错误容器无法解析rancher.rke.com IP：
kubectl -n cattle-system patch  deployments cattle-cluster-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "rancher.rke.com"
                        ],
                            "ip": "172.31.197.183"
                    }
                ]
            }
        }
    }
}'




##########安装harbor例：
安装docker软件包进入docker软件包目录
rpm -ivh *.rpm --nodeps --force
软件包docker-compose 复制到/usr/local/bin/
systemctl enable docker && systemctl start docker
生成harbor证书
mkdir -p /data/cert/harbor/ && cd /data/cert/harbor/
openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 3650 -out ca.crt
openssl req -newkey rsa:4096 -nodes -sha256 -keyout mirror.k8s.com.key -out mirror.k8s.com.csr
openssl x509 -req -days 3650 -in mirror.k8s.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mirror.k8s.com.crt

harbor2.0.tar.gz 解压后配置harbor.yml 设置hostname+证书路径和https port端口
./install.sh

chmod  +x rancher-load-images.sh
上传image到harbor镜像库
./rancher-load-images.sh --image-list ./rancher-images.txt --registry mirror.k8s.com:8989
docker image prune --force --all






##########init初始化脚本
#!/usr/bin/env bash
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
iptables -P FORWARD ACCEPT
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
setenforce 0
sed -ri 's/1/0/' /etc/yum/pluginconf.d/license-manager.conf
echo """
vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
""" > /etc/sysctl.conf
modprobe br_netfilter
sysctl -p
rm -rf /etc/ntp.conf
#cp /root/ntp.conf /etc/ntp.conf
systemctl stop  NetworkManager && systemctl disable NetworkManager && systemctl  disable chronyd && systemctl enable ntpd && systemctl restart ntpd
grub2-set-default "BigCloud Enterprise Linux (4.19.25-200.el7.bclinux.x86_64) 7.                        6 (Core)"
grub2-editenv list



##########新建apps用户脚本
#!/usr/bin/env bash
#创建普通用户并设置密码
useradd -d /home/apps -m apps
echo "123.com" | passwd --stdin apps
#为普通用户赋予sudo权限
chmod u+w /etc/sudoers
echo 'apps    ALL=(ALL)       ALL' >> /etc/sudoers
chmod u-w /etc/sudoers
#禁止普通用户su至root
sed -i 's/^#\(.*required.*\)/\1/' /etc/pam.d/su
echo "SU_WHEEL_ONLY yes" >> /etc/login.defs
#禁止sudo用户修改密码，禁止普通用户关机重启
chmod u+w /etc/sudoers
echo 'apps ALL=/usr/sbin/*,/sbin/*,/usr/bin/*,!/usr/sbin/reboot,!/usr/sbin/poweroff,!/usr/sbin/halt,!/usr/bin/passwd,!/bin/su,!/bin/kill,!/bin/chattr,!/bin/passwd,!/usr/sbin/visudo,!/usr/sbin/useradd,!/usr/sbin/userdel' >> /etc/sudoers
chmod u-w /etc/sudoers
#禁止普通用户修改密码
chmod 511 /usr/bin/passwd
#允许普通用户修改密码
#chmod 4511 /usr/bin/passwd
#所有sudo开个头的文件或文件夹添加i属性（不得任意更动文件或目录）：
chattr +i /etc/sudo.conf
chattr +i /etc/sudoers
chattr +i /etc/sudoers.d/
chattr +i /etc/sudo-ldap.conf



##########clear.sh清理脚本
#!/bin/bash
KUBE_SVC='
kubelet
kube-scheduler
kube-proxy
kube-controller-manager
kube-apiserver
'

for kube_svc in ${KUBE_SVC};
do
  # 停止服务
  if [[ `systemctl is-active ${kube_svc}` == 'active' ]]; then
    systemctl stop ${kube_svc}
  fi
  # 禁止服务开机启动
  if [[ `systemctl is-enabled ${kube_svc}` == 'enabled' ]]; then
    systemctl disable ${kube_svc}
  fi
done

# 停止所有容器
docker stop $(docker ps -aq)

# 删除所有容器
docker rm -f $(docker ps -qa)

# 删除所有容器卷
docker volume rm $(docker volume ls -q)

# 卸载mount目录
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher;
do
  umount $mount;
done

# 备份目录
mv /etc/kubernetes /etc/kubernetes-bak-$(date +"%Y%m%d%H%M")
mv /var/lib/etcd /var/lib/etcd-bak-$(date +"%Y%m%d%H%M")
mv /var/lib/rancher /var/lib/rancher-bak-$(date +"%Y%m%d%H%M")
mv /opt/rke /opt/rke-bak-$(date +"%Y%m%d%H%M")

# 删除残留路径
rm -rf /etc/ceph \
    /etc/cni \
    /opt/cni \
    /run/secrets/kubernetes.io \
    /run/calico \
    /run/flannel \
    /var/lib/calico \
    /var/lib/cni \
    /var/lib/kubelet \
    /var/log/containers \
    /var/log/kube-audit \
    /var/log/pods \
    /var/run/calico

# 清理网络接口
no_del_net_inter='
lo
docker0
eth
ens
bond
'

network_interface=`ls /sys/class/net`

for net_inter in $network_interface;
do
  if ! echo "${no_del_net_inter}" | grep -qE ${net_inter:0:3}; then
    ip link delete $net_inter
  fi
done

# 清理残留进程
port_list='
80
443
6443
2376
2379
2380
8472
9099
10250
10254
'

for port in $port_list;
do
  pid=`netstat -atlnup | grep $port | awk '{print $7}' | awk -F '/' '{print $1}' | grep -v - | sort -rnk2 | uniq`
  if [[ -n $pid ]]; then
    kill -9 $pid
  fi
done

kube_pid=`ps -ef | grep -v grep | grep kube | awk '{print $2}'`

if [[ -n $kube_pid ]]; then
  kill -9 $kube_pid
fi

# 清理Iptables表
## 注意：如果节点Iptables有特殊配置，以下命令请谨慎操作
sudo iptables --flush
sudo iptables --flush --table nat
sudo iptables --flush --table filter
sudo iptables --table nat --delete-chain
sudo iptables --table filter --delete-chain
systemctl restart docker





##########新集群master-恢复kubeconfig文件脚本（restore-kube-config.sh）

#!/bin/bash

help ()
{
echo ' ================================================================ '
echo ' --master-ip: 指定 Master 节点 IP，任意一个 K8S Master 节点 IP 即可。'
echo ' 使用示例：bash restore-kube-config.sh --master-ip=1.1.1.1 '
echo ' ================================================================'
}

case "$1" in
-h|--help) help; exit;;
esac

if [[ $1 == '' ]]; then
help;
exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
case "$key" in
--master-ip) K8S_MASTER_NODE_IP=$value ;;
esac
done

# 获取 Rancher Agent 镜像
RANCHER_IMAGE=$( docker images --filter=label=io.cattle.agent=true |grep 'v2.' | \
grep -v -E 'rc|alpha|<none>' | head -n 1 | awk '{print $3}' )

if [[ -d /etc/kubernetes/ssl ]]; then
K8S_SSLDIR=/etc/kubernetes/ssl
else
echo '/etc/kubernetes/ssl 目录不存在'
exit 1
fi

CHECK_CLUSTER_STATE_CONFIGMAP=$( docker run --rm --entrypoint bash --net=host \
-v $K8S_SSLDIR:/etc/kubernetes/ssl:ro $RANCHER_IMAGE -c '\
if kubectl --kubeconfig /etc/kubernetes/ssl/kubecfg-kube-node.yaml \
-n kube-system get configmap full-cluster-state | grep full-cluster-state > /dev/null; then \
echo 'yes'; else echo 'no'; fi' )

if [[ $CHECK_CLUSTER_STATE_CONFIGMAP != 'yes' ]]; then

docker run --rm --net=host \
--entrypoint bash \
-e K8S_MASTER_NODE_IP=$K8S_MASTER_NODE_IP \
-v $K8S_SSLDIR:/etc/kubernetes/ssl:ro \
$RANCHER_IMAGE \
-c '\
kubectl --kubeconfig /etc/kubernetes/ssl/kubecfg-kube-node.yaml \
-n kube-system \
get secret kube-admin -o jsonpath={.data.Config} | base64 --decode | \
sed -e "/^[[:space:]]*server:/ s_:.*_: \"https://${K8S_MASTER_NODE_IP}:6443\"_"' > kubeconfig_admin.yaml

if [[ -s kubeconfig_admin.yaml ]]; then
echo '恢复成功，执行以下命令测试：'
echo ''
echo "kubectl --kubeconfig kubeconfig_admin.yaml get nodes"
else
echo "kubeconfig 恢复失败。"
fi
fi