rancher 在三台机器搭建k8s集群

一、基础环境说明

节点名	节点ipd	角色	操作系统
node1	10.42.8.13	control-plane,etcd,master	CentOS7.9
node2	10.42.8.14	control-plane,etcd,master	CentOS7.9
node3	10.42.8.15	control-plane,etcd,master	CentOS7.9
二、k8s节点机基础环境设置
1、设置 hostname（三台节点分别执行）

# node1
hostnamectl set-hostname node1
# node2
hostnamectl set-hostname node2
# node3
hostnamectl set-hostname node3

2、设置 /etc/hosts（三台节点都执行）

cat <<EOF > /etc/hosts
10.42.8.13 node1
10.42.8.14 node2
10.42.8.15 node3
EOF

3、设置 iptables（三台节点都执行）

iptables -P FORWARD ACCEPT

4、关闭 swap（三台节点都执行）

swapoff -a
# 防止开机自动挂载 swap 分区（将 /etc/fstab 中的内容注释掉）
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

5、关闭 selinux（三台节点都执行）

# 将 selinux 由 enforcing 改为 disabled 。注意下面命令中 disabled 前面是数字 1，不是小写的英文字母 l
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
# 临时关闭 enforce
setenforce 0
# 验证 enforce 是否已关闭：如关闭，应返回结果：Disabled
getenforce

6、将 NetworkManager 配置为忽略 calico/flannel 相关的网络接口（三台节点都执行）

cat > /etc/NetworkManager/conf.d/rke2-canal.conf <<EOF
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:flannel*
EOF

6、关闭并禁用firewalld和NetworkManager（三台节点都执行）

systemctl stop firewalld && systemctl disable firewalld
systemctl stop NetworkManager && systemctl disable NetworkManager

7、修改内核参数（三台节点都执行）

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.max_map_count = 262144
vm.swappiness=0
EOF

三、rke2部署k8s
参考文档：https://ranchermanager.docs.rancher.com/zh/how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher

1、配置rke2配置文件

说明：token可自定义字符串，但注意所有节点的token要一样

在node1上执行

mkdir -p /etc/rancher/rke2/
cat > /etc/rancher/rke2/config.yaml <<EOF
token: K108a91d54650d8ggiegn923ijgg2a9be9d1483fc932e2221f0750f14::server:57f9a32eee46999c96e936927d09af9b
tls-san: 10.42.8.13
system-default-registry: "registry.cn-hangzhou.aliyuncs.com"
cluster-cidr: 10.251.0.0/16
service-cidr: 10.252.0.0/16
 
ingress:
  provider: nginx
  options:
    config-snipper: |
      # 启用 snippet
      enable-snippet-annotation: "true"
EOF

在node2和node3上执行

mkdir -p /etc/rancher/rke2/
cat > /etc/rancher/rke2/config.yaml <<EOF
server: https://10.42.8.13:9345
token: K108a91d54650d8ggiegn923ijgg2a9be9d1483fc932e2221f0750f14::server:57f9a32eee46999c96e936927d09af9b
tls-san: 10.42.8.13
system-default-registry: "registry.cn-hangzhou.aliyuncs.com"
cluster-cidr: 10.251.0.0/16
service-cidr: 10.252.0.0/16
EOF

2、安装并启用rke2（三个节点都执行）。特别说明：需要等node1启动成功rke2-server后，才能在node2和node3上执行

curl -sfL https://rancher-mirror.rancher.cn/rke2/install.sh | INSTALL_RKE2_MIRROR=cn INSTALL_RKE2_VERSION=v1.25.12+rke2r1 sh -
systemctl enable rke2-server && systemctl start rke2-server

3、软连接集群配置文件和操作工具（只需在node1上执行）

ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
ln -s /var/lib/rancher/rke2/bin/crictl /usr/local/bin/crictl
ln -s /var/lib/rancher/rke2/bin/ctr /usr/local/bin/ctr
ln -s /etc/rancher/rke2/rke2.yaml ~/.kube/config

4、设置crictl的默认socket（只需在node1上执行）

说明：这里是rke2的一个bug，默认设置的socket是unix:///run/containerd/containerd.sock，而实际的确是：unix:///run/k3s/containerd/containerd.sock

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/k3s/containerd/containerd.sock
image-endpoint: unix:///run/k3s/containerd/containerd.sock
timeout: 10
debug: false
EOF

5、crt工具使用时，先设置alias别名

说明：这里是rke2的一个bug，默认设置的socket是unix:///run/containerd/containerd.sock，而实际的确是：unix:///run/k3s/containerd/containerd.sock

cat >> ~/.bashrc <<EOF
alias ctr='ctr --address=/run/k3s/containerd/containerd.sock'
EOF
 
source ~/.bashrc

6、至此，k8s部署完成，检查node和pod是否正常运行

[root@node1 ~]# kubectl get no
NAME                          STATUS   ROLES                       AGE   VERSION
node1   Ready    control-plane,etcd,master   22d   v1.25.12+rke2r1
node2   Ready    control-plane,etcd,master   21d   v1.25.12+rke2r1
node3   Ready    control-plane,etcd,master   21d   v1.25.12+rke2r1
[root@node1 ~]#
[root@node1 ~]# kubectl get pod -A
NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-rke2-server-1                  1/1     Running     0          2m28s
kube-system   cloud-controller-manager-rke2-server-2                  1/1     Running     0          61s
kube-system   cloud-controller-manager-rke2-server-3                  1/1     Running     0          49s
kube-system   etcd-rke2-server-1                                      1/1     Running     0          2m13s
kube-system   etcd-rke2-server-2                                      1/1     Running     0          87s
kube-system   etcd-rke2-server-3                                      1/1     Running     0          56s
kube-system   helm-install-rke2-canal-hs6sx                           0/1     Completed   0          2m17s
kube-system   helm-install-rke2-coredns-xmzm8                         0/1     Completed   0          2m17s
kube-system   helm-install-rke2-ingress-nginx-flwnl                   0/1     Completed   0          2m17s
kube-system   helm-install-rke2-metrics-server-7sggn                  0/1     Completed   0          2m17s
kube-system   kube-apiserver-rke2-server-1                            1/1     Running     0          116s
kube-system   kube-apiserver-rke2-server-2                            1/1     Running     0          66s
kube-system   kube-apiserver-rke2-server-3                            1/1     Running     0          48s
kube-system   kube-controller-manager-rke2-server-1                   1/1     Running     0          2m30s
kube-system   kube-controller-manager-rke2-server-2                   1/1     Running     0          57s
kube-system   kube-controller-manager-rke2-server-3                   1/1     Running     0          42s
kube-system   kube-proxy-rke2-server-1                                1/1     Running     0          2m25s
kube-system   kube-proxy-rke2-server-2                                1/1     Running     0          59s
kube-system   kube-proxy-rke2-server-3                                1/1     Running     0          85s
kube-system   kube-scheduler-rke2-server-1                            1/1     Running     0          2m30s
kube-system   kube-scheduler-rke2-server-2                            1/1     Running     0          57s
kube-system   kube-scheduler-rke2-server-3                            1/1     Running     0          42s
kube-system   rke2-canal-b9lvm                                        2/2     Running     0          91s
kube-system   rke2-canal-khwp2                                        2/2     Running     0          2m5s
kube-system   rke2-canal-swfmq                                        2/2     Running     0          105s
kube-system   rke2-coredns-rke2-coredns-547d5499cb-6tvwb              1/1     Running     0          92s
kube-system   rke2-coredns-rke2-coredns-547d5499cb-rdttj              1/1     Running     0          2m8s
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-85sq5   1/1     Running     0          2m8s
kube-system   rke2-ingress-nginx-controller-69qxc                     1/1     Running     0          52s
kube-system   rke2-ingress-nginx-controller-7hprp                     1/1     Running     0          52s
kube-system   rke2-ingress-nginx-controller-x658h                     1/1     Running     0          52s
kube-system   rke2-metrics-server-6564db4569-vdfkn                    1/1     Running     0          66s

四、安装 Rancher Helm Chart（仅在node1上执行）
1、安装helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

2、添加 Helm Chart 仓库

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

3、为 Rancher 创建命名空间

kubectl create namespace cattle-system

4、添加rancher所需的secret，注意秘钥文件路径，可自定义，需要先把秘钥文件上传到对应的目录下

kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=/opt/work/ssl_cert/skyeye.crt --key=/opt/work/ssl_cert/skyeye.key

5、通过 Helm 安装 Rancher

helm install rancher rancher-stable/rancher --namespace cattle-system --version v2.7.5 --set hostname=rancher.platform.com --set bootstrapPassword=admin --set ingress.tls.source=secret

6、至此，rancher部署完成，rancher登录页为：https://rancher.platform.com