二进制 k8s集群（docker）

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#client-binaries 官方二进制下载站点

一、环境

节点信息

本环境使用阿里云，API Server 高可用通过阿里云SLB实现，如果环境不在云上，可以通过 Nginx + Keepalived，或者 HaProxy + Keepalived等实现。

服务版本与K8S集群说明

阿里slb设置TCP监听，监听6443端口(通过四层负载到master apiserver)。
所有阿里云ECS主机使用 CentOS 7.6.1810 版本，并且内核都升到5.x版本。
K8S 集群使用 Iptables 模式（kube-proxy 注释中预留 Ipvs 模式配置）
Calico 使用 IPIP 模式
集群使用默认 svc.cluster.local
10.10.0.1 为集群 kubernetes svc 解析ip
Docker CE version 19.03.6
Kubernetes Version 1.18.2
Etcd Version v3.4.7
Calico Version v3.14.0
Coredns Version 1.6.7
Metrics-Server Version v0.3.6

Service 和 Pods Ip 段划分

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.10.0.1    <none>        443/TCP   6d23h

Service 和 Pods Ip 段划分

二、环境初始化

所有集群节点都需要初始化

2.1 停止所有机器 firewalld 防火墙

$ systemctl stop firewalld
$ systemctl disable firewalld

2.2 关闭 swap

$ swapoff -a 
$ sed -i 's/.*swap.*/#&/' /etc/fstab

2.3 关闭 Selinux

$ setenforce  0 
$ sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
$ sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
$ sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
$ sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config

2.4 设置主机名、升级内核、安装 Docker ce

运行下面 init.sh shell 脚本，脚本完成下面四项任务：

设置服务器 hostname
安装 k8s依赖环境
升级系统内核（升级Centos7系统内核，解决Docker-ce版本兼容问题）
安装 docker ce 19.03.6 版本

在每台机器上运行 init.sh 脚本，示例如下：

Ps：init.sh 脚本只用于 Centos，支持 重复运行。

# k8s-master1 机器运行，init.sh 后面接的参数是设置 k8s-master1 服务器主机名
$ chmod +x init.sh && ./init.sh k8s-master1

# 执行完 init.sh 脚本，请重启服务器
$ reboot

#!/usr/bin/bash

function Check_linux_system(){
    linux_version=`cat /etc/redhat-release`
    if [[ ${linux_version} =~ "CentOS" ]];then
        echo -e "\033[32;32m 系统为 ${linux_version} \033[0m \n"
    else
        echo -e "\033[32;32m 系统不是CentOS,该脚本只支持CentOS环境\033[0m \n"
        exit 1
    fi
}

function Set_hostname(){
    if [ -n "$HostName" ];then
      grep $HostName /etc/hostname && echo -e "\033[32;32m 主机名已设置，退出设置主机名步骤 \033[0m \n" && return
      case $HostName in
      help)
        echo -e "\033[32;32m bash init.sh 主机名 \033[0m \n"
        exit 1
      ;;
      *)
        hostname $HostName
        echo "$HostName" > /etc/hostname
        echo "`ifconfig eth0 | grep inet | awk '{print $2}'` $HostName" >> /etc/hosts
      ;;
      esac
    else
      echo -e "\033[32;32m 输入为空，请参照 bash init.sh 主机名 \033[0m \n"
      exit 1
    fi
}

function Install_depend_environment(){
    rpm -qa | grep nfs-utils &> /dev/null && echo -e "\033[32;32m 已完成依赖环境安装，退出依赖环境安装步骤 \033[0m \n" && return
    yum install -y nfs-utils curl yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl telnet
    echo -e "\033[32;32m 升级Centos7系统内核到5版本，解决Docker-ce版本兼容问题\033[0m \n"
    rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org && \
    rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm && \
    yum --disablerepo=\* --enablerepo=elrepo-kernel repolist && \
    yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-ml.x86_64 && \
    yum remove -y kernel-tools-libs.x86_64 kernel-tools.x86_64 && \
    yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-ml-tools.x86_64 && \
    grub2-set-default 0
    modprobe br_netfilter
    cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
    sysctl -p /etc/sysctl.d/k8s.conf
    ls /proc/sys/net/bridge
}

function Install_docker(){
    rpm -qa | grep docker && echo -e "\033[32;32m 已安装docker，退出安装docker步骤 \033[0m \n" && return
    yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    yum makecache fast
    yum -y install docker-ce-19.03.6 docker-ce-cli-19.03.6
    # 设置 iptables file表中 FORWARD 默认链规则为 ACCEPT
    sed  -i '/ExecStart=/i ExecStartPost=\/sbin\/iptables -P FORWARD ACCEPT' /usr/lib/systemd/system/docker.service
    systemctl enable docker.service
    systemctl start docker.service
    systemctl stop docker.service
    echo '{"registry-mirrors": ["https://4xr1qpsp.mirror.aliyuncs.com"], "log-opts": {"max-size":"500m", "max-file":"3"}}' > /etc/docker/daemon.json
    systemctl daemon-reload
    systemctl start docker
}

# 初始化顺序
HostName=$1
Check_linux_system && \
Set_hostname && \
Install_depend_environment && \
Install_docker

三、Kubernetes 部署

部署顺序

1、自签TLS证书
2、部署Etcd集群
3、创建 metrics-server 证书
4、获取K8S二进制包
5、创建Node节点kubeconfig文件
6、配置Master组件并运行
7、配置kubelet证书自动续期和创建Node授权用户
8、配置Node组件并运行
9、安装calico网络，使用IPIP模式
10、集群CoreDNS部署
11、部署集群监控服务 Metrics Server
12、部署 Kubernetes Dashboard

3.1 自签TLS证书

在 k8s-master1 安装证书生成工具 cfssl，并生成相关证书

# 创建目录用于存放 SSL 证书
$ mkdir /data/ssl -p

# 下载生成证书命令
$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

# 添加执行权限
$ chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

# 移动到 /usr/local/bin 目录下
$ mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
$ mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

# 进入证书目录
$ cd /data/ssl/

# 创建 certificate.sh 脚本
$ vim  certificate.sh

PS：证书有效期为 10年

脚本如下：根据自己的环境修改 certificate.sh 脚本

 "192.168.0.216",
 "192.168.0.217",
 "192.168.0.218",
 "10.10.0.1",
 "lb.ypvip.com.cn",

 修改完脚本，然后执行

$ bash certificate.sh

# cat certificate.sh

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
              "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------

cat > server-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.0.216",
      "192.168.0.217",
      "192.168.0.218",
      "10.10.0.1",
      "lb.ypvip.com.cn",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

#-----------------------

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#-----------------------

cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

3.2 部署Etcd集群

k8s-master1 机器上操作，把执行文件copy到 k8s-master2 k8s-master3

二进制包下载地址：https://github.com/etcd-io/etcd/releases/download/v3.4.7/etcd-v3.4.7-linux-amd64.tar.gz

# 创建存储etcd数据目录
$ mkdir /data/etcd/

# 创建 k8s 集群配置目录
$ mkdir /opt/kubernetes/{bin,cfg,ssl}  -p

# 下载二进制etcd包，并把执行文件放到 /opt/kubernetes/bin/ 目录
$ cd /data/etcd/
$ wget https://github.com/etcd-io/etcd/releases/download/v3.4.7/etcd-v3.4.7-linux-amd64.tar.gz
$ tar zxvf etcd-v3.4.7-linux-amd64.tar.gz
$ cd etcd-v3.4.7-linux-amd64
$ cp -a etcd etcdctl /opt/kubernetes/bin/

# 把 /opt/kubernetes/bin 目录加入到 PATH
$ echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile
$ source /etc/profile

登陆到 k8s-master2 和 k8s-master3 服务器上操作

# 创建 k8s 集群配置目录
$ mkdir /data/etcd
$ mkdir /opt/kubernetes/{bin,cfg,ssl}  -p

# 把 /opt/kubernetes/bin 目录加入到 PATH
$ echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile
$ source /etc/profile

登陆到 k8s-master1 操作

# 进入 K8S 集群证书目录
$ cd /data/ssl

# 把证书 copy 到 k8s-master1 机器 /opt/kubernetes/ssl/ 目录
$ cp ca*pem  server*pem /opt/kubernetes/ssl/

# 把etcd执行文件与证书 copy 到 k8s-master2  k8s-master3 机器 
scp -r /opt/kubernetes/*  root@k8s-master2:/opt/kubernetes
scp -r /opt/kubernetes/*  root@k8s-master3:/opt/kubernetes

$ cd /data/etcd

# 编写 etcd 配置文件脚本
$ vim etcd.sh
#!/bin/bash

ETCD_NAME=${1:-"etcd01"}
ETCD_IP=${2:-"127.0.0.1"}
ETCD_CLUSTER=${3:-"etcd01=https://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/etcd.yml
name: ${ETCD_NAME}
data-dir: /var/lib/etcd/default.etcd
listen-peer-urls: https://${ETCD_IP}:2380
listen-client-urls: https://${ETCD_IP}:2379,https://127.0.0.1:2379

advertise-client-urls: https://${ETCD_IP}:2379
initial-advertise-peer-urls: https://${ETCD_IP}:2380
initial-cluster: ${ETCD_CLUSTER}
initial-cluster-token: etcd-cluster
initial-cluster-state: new

client-transport-security:
  cert-file: /opt/kubernetes/ssl/server.pem
  key-file: /opt/kubernetes/ssl/server-key.pem
  client-cert-auth: false
  trusted-ca-file: /opt/kubernetes/ssl/ca.pem
  auto-tls: false

peer-transport-security:
  cert-file: /opt/kubernetes/ssl/server.pem
  key-file: /opt/kubernetes/ssl/server-key.pem
  client-cert-auth: false
  trusted-ca-file: /opt/kubernetes/ssl/ca.pem
  auto-tls: false

debug: false
logger: zap
log-outputs: [stderr]
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/etcd-io/etcd
Conflicts=etcd.service
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
LimitNOFILE=65536
Restart=on-failure
RestartSec=5s
TimeoutStartSec=0
ExecStart=/opt/kubernetes/bin/etcd --config-file=/opt/kubernetes/cfg/etcd.yml

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

# 执行 etcd.sh 生成配置脚本
$ chmod +x etcd.sh
$ ./etcd.sh etcd01 192.168.0.216 etcd01=https://192.168.0.216:2380,etcd02=https://192.168.0.217:2380,etcd03=https://192.168.0.218:2380

# 查看 etcd 是否启动正常
$ ps -ef | grep etcd 
$ netstat -ntplu | grep etcd

tcp        0      0 192.168.0.216:2379      0.0.0.0:*               LISTEN      1558/etcd
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      1558/etcd
tcp        0      0 192.168.0.216:2380      0.0.0.0:*               LISTEN      1558/etcd

# 把 etcd.sh 脚本  copy 到 k8s-master2 k8s-master3 机器上
$ scp /data/etcd/etcd.sh  root@k8s-master2:/data/etcd/
$ scp /data/etcd/etcd.sh  root@k8s-master3:/data/etcd/

登陆到 k8s-master2 操作

# 执行 etcd.sh 生成配置脚本
$ chmod +x etcd.sh
$ ./etcd.sh etcd02 192.168.0.217 etcd01=https://192.168.0.216:2380,etcd02=https://192.168.0.217:2380,etcd03=https://192.168.0.218:2380

# 查看 etcd 是否启动正常
$ ps -ef | grep etcd 
$ netstat -ntplu | grep etcd

登陆到 k8s-master3 操作

# 执行 etcd.sh 生成配置脚本
$ chmod +x etcd.sh
$ ./etcd.sh etcd03 192.168.0.218 etcd01=https://192.168.0.216:2380,etcd02=https://192.168.0.217:2380,etcd03=https://192.168.0.218:2380

# 查看 etcd 是否启动正常
$ ps -ef | grep etcd 
$ netstat -ntplu | grep etcd

# 随便登陆一台master机器，查看 etcd 集群是否正常
$ ETCDCTL_API=3 etcdctl --write-out=table \
--cacert=/opt/kubernetes/ssl/ca.pem --cert=/opt/kubernetes/ssl/server.pem --key=/opt/kubernetes/ssl/server-key.pem \
--endpoints=https://192.168.0.216:2379,https://192.168.0.217:2379,https://192.168.0.218:2379 endpoint health

+---------------------------------+--------+-------------+-------+
|            ENDPOINT             | HEALTH |    TOOK     | ERROR |
+---------------------------------+--------+-------------+-------+
| https://192.168.0.216:2379      |   true | 38.721248ms |       |
| https://192.168.0.217:2379      |   true | 38.621248ms |       |
| https://192.168.0.218:2379      |   true | 38.821248ms |       |
+---------------------------------+--------+-------------+-------+

3.3 创建 metrics-server 证书

创建 metrics-server 使用的证书

登陆到 k8s-master1 操作

$ cd /data/ssl/

# 注意: "CN": "system:metrics-server" 一定是这个，因为后面授权时用到这个名称，否则会报禁止匿名访问
$ cat > metrics-server-csr.json <<EOF
{
  "CN": "system:metrics-server",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

生成 metrics-server 证书和私钥

# 生成证书
$ cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server

# copy 到 /opt/kubernetes/ssl 目录
$ cp metrics-server-key.pem metrics-server.pem /opt/kubernetes/ssl/

# copy 到 k8s-master2 k8s-master3 机器上
$ scp metrics-server-key.pem metrics-server.pem root@k8s-master2:/opt/kubernetes/ssl/
$ scp metrics-server-key.pem metrics-server.pem root@k8s-master3:/opt/kubernetes/ssl/

3.4 获取K8S二进制包

登陆到 k8s-master1 操作

v1.18 下载页面 
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md


# 创建存放 k8s 二进制包目录
$ mkdir /data/k8s-package
$ cd /data/k8s-package

# 下载 v1.18.2 二进制包
# 作者把二进制安装包上传到cdn上 https://cdm.yp14.cn/k8s-package/kubernetes-server-v1.18.2-linux-amd64.tar.gz
$ wget https://dl.k8s.io/v1.18.2/kubernetes-server-linux-amd64.tar.gz

$ tar xf kubernetes-server-linux-amd64.tar.gz

master 节点需要用到：

    kubectl
    kube-scheduler
    kube-apiserver
    kube-controller-manager

node 节点需要用到：

    kubelet
    kube-proxy

PS：本文master节点也做为一个node节点，所以需要用到 kubelet kube-proxy 执行文件

# 进入解压出来二进制包bin目录
$ cd /data/k8s-package/kubernetes/server/bin

# cpoy 执行文件到 /opt/kubernetes/bin 目录
$ cp -a kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy /opt/kubernetes/bin

# copy 执行文件到 k8s-master2 k8s-master3 机器 /opt/kubernetes/bin 目录
$ scp kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy root@k8s-master2:/opt/kubernetes/bin/
$ scp kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy root@k8s-master3:/opt/kubernetes/bin/

3.5 创建Node节点kubeconfig文件

登陆到 k8s-master1 操作

创建TLS Bootstrapping Token
创建kubelet kubeconfig
创建kube-proxy kubeconfig

$ cd /data/ssl/

# 修改第10行 KUBE_APISERVER 地址
$ vim kubeconfig.sh

修改 kubeconfig.sh 脚本配置 KUBE_APISERVER 变量时，一定要把 https:// 带上，否则默认使用 http:// 请求

# 创建 TLS Bootstrapping Token
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

#----------------------

# 创建kubelet bootstrapping kubeconfig
# 一定要把 https:// 带上，否则默认使用 http:// 请求
export KUBE_APISERVER="https://lb.ypvip.com.cn:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 创建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

# 生成证书
$ sh kubeconfig.sh

# 输出下面结果
kubeconfig.sh   kube-proxy-csr.json  kube-proxy.kubeconfig
kube-proxy.csr  kube-proxy-key.pem   kube-proxy.pem bootstrap.kubeconfig

# copy *kubeconfig 文件到 /opt/kubernetes/cfg 目录
$ cp *kubeconfig /opt/kubernetes/cfg

# copy 到 k8s-master2 k8s-master3 机器上
$ scp *kubeconfig root@k8s-master2:/opt/kubernetes/cfg
$ scp *kubeconfig root@k8s-master3:/opt/kubernetes/cfg

3.6 配置Master组件并运行

登陆到 k8s-master1 k8s-master2 k8s-master3 操作

# 创建 /data/k8s-master 目录，用于存放 master 配置执行脚本
$ mkdir /data/k8s-master

# 创建 kube-apiserver 日志存放目录
$ mkdir -p /var/log/kubernetes

# 创建 kube-apiserver 审计日志文件
$ touch /var/log/kubernetes/k8s-audit.log

登陆到 k8s-master1

$ cd /data/k8s-master

# 创建生成 kube-apiserver 配置文件脚本

$ vim apiserver.sh
#!/bin/bash

MASTER_ADDRESS=${1:-"192.168.0.216"}
ETCD_SERVERS=${2:-"http://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/var/log/kubernetes \\
--etcd-servers=${ETCD_SERVERS} \\
--bind-address=0.0.0.0 \\
--secure-port=6443 \\
--advertise-address=${MASTER_ADDRESS} \\
--allow-privileged=true \\
--service-cluster-ip-range=10.10.0.0/16 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--kubelet-https=true \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-50000 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \\
--etcd-certfile=/opt/kubernetes/ssl/server.pem \\
--etcd-keyfile=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--proxy-client-cert-file=/opt/kubernetes/ssl/metrics-server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/metrics-server-key.pem \\
--runtime-config=api/all=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-truncate-enabled=true \\
--audit-log-path=/var/log/kubernetes/k8s-audit.log"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver

# 创建生成 kube-controller-manager 配置文件脚本
$ vim controller-manager.sh
#!/bin/bash

MASTER_ADDRESS=${1:-"127.0.0.1"}

cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\
--v=2 \\
--master=${MASTER_ADDRESS}:8080 \\
--leader-elect=true \\
--bind-address=0.0.0.0 \\
--service-cluster-ip-range=10.10.0.0/16 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s \\
--feature-gates=RotateKubeletServerCertificate=true \\
--feature-gates=RotateKubeletClientCertificate=true \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.20.0.0/16 \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager

# 创建生成 kube-scheduler 配置文件脚本
$ vim scheduler.sh
#!/bin/bash

MASTER_ADDRESS=${1:-"127.0.0.1"}

cat <<EOF >/opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \\
--v=2 \\
--master=${MASTER_ADDRESS}:8080 \\
--address=0.0.0.0 \\
--leader-elect"
EOF

cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler

# 添加执行权限
$ chmod +x *.sh

$ cp /data/ssl/token.csv /opt/kubernetes/cfg/

# copy token.csv 和 master 配置到 k8s-master2 k8s-master3 机器上
$ scp /data/ssl/token.csv root@k8s-master2:/opt/kubernetes/cfg
$ scp /data/ssl/token.csv root@k8s-master3:/opt/kubernetes/cfg
$ scp apiserver.sh controller-manager.sh scheduler.sh root@k8s-master2:/data/k8s-master
$ scp apiserver.sh controller-manager.sh scheduler.sh root@k8s-master3:/data/k8s-master

# 生成 master配置文件并运行
$ ./apiserver.sh 192.168.0.216 https://192.168.0.216:2379,https://192.168.0.217:2379,https://192.168.0.218:2379 
$ ./controller-manager.sh 127.0.0.1
$ ./scheduler.sh 127.0.0.1

# 查看master三个服务是否正常运行
$ ps -ef | grep kube
$ netstat -ntpl | grep kube-

登陆到 k8s-master2 操作

$ cd /data/k8s-master

# 生成 master配置文件并运行
$ ./apiserver.sh 192.168.0.217 https://192.168.0.216:2379,https://192.168.0.217:2379,https://192.168.0.218:2379 
$ ./controller-manager.sh 127.0.0.1
$ ./scheduler.sh 127.0.0.1

# 查看master三个服务是否正常运行
$ ps -ef | grep kube
$ netstat -ntpl | grep kube-

登陆到 k8s-master3 操作

$ cd /data/k8s-master

# 生成 master配置文件并运行
$ ./apiserver.sh 192.168.0.218 https://192.168.0.216:2379,https://192.168.0.217:2379,https://192.168.0.218:2379 
$ ./controller-manager.sh 127.0.0.1
$ ./scheduler.sh 127.0.0.1

# 查看master三个服务是否正常运行
$ ps -ef | grep kube
$ netstat -ntpl | grep kube-

# 随便登陆一台master查看集群健康状态
$ kubectl  get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-2               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-0               Healthy   {"health":"true"}

错误处理

# 查询集群状态报下面错误
$ kubectl get cs

error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

两种解决方法：

1、可以把kubectl版本降级到1.17版本，生成管理员config文件，并存放到 ~/.kube/config 目录下，再把 kubectl升级到1.18版本，这时就可以正常使用
2、直接声明下 KUBERNETES_MASTER Apiserver 地址，不推荐这种方法

这里介绍生成管理员 config 配置

# 降级 kubectl 版本到 1.17.5 
$ mv /opt/kubernetes/bin/kubectl /opt/kubernetes/bin/kubectl-1.18.2
$ wget https://cdm.yp14.cn/k8s-package/k8s-1.17-bin/kubectl -O /opt/kubernetes/bin/kubectl
$ chmod +x /opt/kubernetes/bin/kubectl

# 创建一个存放用户文件目录
$ mkdir -p /root/yaml/create-user
$ cd /root/yaml/create-user

# Copy 脚本依赖文件
$ cp /data/ssl/ca-config.json /opt/kubernetes/ssl/

# 创建一个生成用户脚本
$ vim create-user-kubeconfig.sh
#!/bin/bash
# 注意修改KUBE_APISERVER为你的API Server的地址

KUBE_APISERVER=$1
USER=$2
USER_SA=system:serviceaccount:default:${USER}
Authorization=$3
USAGE="USAGE: create-user.sh <api_server> <username> <clusterrole authorization>\n
Example: https://lb.ypvip.com.cn:6443 brand"
CSR=`pwd`/user-csr.json
SSL_PATH="/opt/kubernetes/ssl"
USER_SSL_PATH="/root/yaml/create-user"
SSL_FILES=(ca-key.pem ca.pem ca-config.json)
CERT_FILES=(${USER}.csr $USER-key.pem ${USER}.pem)

if [[ $KUBE_APISERVER == "" ]]; then
   echo -e $USAGE
   exit 1
fi
if [[ $USER == "" ]];then
    echo -e $USAGE
    exit 1
fi

if [[ $Authorization == "" ]];then
    echo -e $USAGE
    exit 1
fi

# 创建用户的csr文件
function createCSR(){
cat>$CSR<<EOF
{
  "CN": "USER",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
# 替换csr文件中的用户名
sed -i "s/USER/$USER_SA/g" $CSR
}

function ifExist(){
if [ ! -f "$SSL_PATH/$1" ]; then
    echo "$SSL_PATH/$1 not found."
    exit 1
fi
}

function ifClusterrole(){
kubectl get clusterrole ${Authorization} &> /dev/null
if (( $? !=0 ));then
   echo "${Authorization} clusterrole there is no"
   exit 1
fi
}

# 判断clusterrole授权是否存在
ifClusterrole

# 判断证书文件是否存在
for f in ${SSL_FILES[@]};
do
    echo "Check if ssl file $f exist..."
    ifExist $f
    echo "OK"
done

echo "Create CSR file..."
createCSR
echo "$CSR created"
echo "Create user's certificates and keys..."
cd $USER_SSL_PATH
cfssl gencert -ca=${SSL_PATH}/ca.pem -ca-key=${SSL_PATH}/ca-key.pem -config=${SSL_PATH}/ca-config.json -profile=kubernetes $CSR| cfssljson -bare $USER_SA

# 创建 sa
kubectl create sa ${USER} -n default

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=${SSL_PATH}/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${USER}.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials ${USER_SA} \
--client-certificate=${USER_SSL_PATH}/${USER_SA}.pem \
--client-key=${USER_SSL_PATH}/${USER_SA}-key.pem \
--embed-certs=true \
--kubeconfig=${USER}.kubeconfig

# 设置上下文参数
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=${USER_SA} \
--namespace=default \
--kubeconfig=${USER}.kubeconfig

# 设置默认上下文
kubectl config use-context kubernetes --kubeconfig=${USER}.kubeconfig

# 创建 namespace
# kubectl create ns $USER

# 绑定角色
# kubectl create rolebinding ${USER}-admin-binding --clusterrole=admin --user=$USER --namespace=$USER --serviceaccount=$USER:default
kubectl create clusterrolebinding ${USER}-binding --clusterrole=${Authorization} --user=${USER_SA}

# kubectl config get-contexts

echo "Congratulations!"
echo "Your kubeconfig file is ${USER}.kubeconfig"

# 添加执行权限
$ chmod +x create-user-kubeconfig.sh

# 脚本 Help
$ ./create-user-kubeconfig.sh --help

USAGE: create-user.sh <api_server> <username> <clusterrole authorization>
 Example: https://lb.ypvip.com.cn:6443 brand

# 创建一个 admin 管理员用户，绑定 cluster-admin 权限
$ ./create-user-kubeconfig.sh https://lb.ypvip.com.cn:6443 admin cluster-admin

# 查看 admin 用户 Token
$ kubectl describe secrets -n default `kubectl  get secrets -n default | grep admin-token | awk '{print $1}'` | grep 'token:'

# 把上面查看的 Token 写入到脚本生成 admin.kubeconfig 文件最底部
$ vim admin.kubeconfig

# 创建 .kube 目录
$ mkdir ~/.kube/

# 拷贝到 .kube 目录下
$ cp admin.kubeconfig ~/.kube/config

# 把kubectl 从 1.17.5 换成 1.18.2版本，可以正常查看集群状态
$ mv /opt/kubernetes/bin/kubectl-1.18.2 /opt/kubernetes/bin/kubectl
$ chmod +x /opt/kubernetes/bin/kubectl
$ kubectl  get cs

NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-2               Healthy   {"health":"true"}

3.7 配置kubelet证书自动续期和创建Node授权用户

登陆到 k8s-master1 操作

创建 Node节点 授权用户 kubelet-bootstrap

$ kubectl create clusterrolebinding  kubelet-bootstrap --clusterrole=system:node-bootstrapper  --user=kubelet-bootstrap

创建自动批准相关 CSR 请求的 ClusterRole

# 创建证书旋转配置存放目录
$ mkdir ~/yaml/kubelet-certificate-rotating
$ cd ~/yaml/kubelet-certificate-rotating

$ vim tls-instructs-csr.yaml 
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]

# 部署
$ kubectl apply -f tls-instructs-csr.yaml

自动批准 kubelet-bootstrap 用户 TLS bootstrapping 首次申请证书的 CSR 请求

$ kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --user=kubelet-bootstrap

自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求

$ kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes

自动批准 system:nodes 组用户更新 kubelet 10250 api 端口证书的 CSR 请求

$ kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes

3.8 配置Node组件并运行

首先我们先了解下 kubelet 中 kubelet.kubeconfig 配置是如何生成？

kubelet.kubeconfig 配置是通过 TLS Bootstrapping 机制生成，下面是生成的流程图。

登陆到 k8s-master1 k8s-master2 k8s-master3 操作

# 创建 node 节点生成配置脚本目录
$ mkdir /data/k8s-node

登陆到 k8s-master1 操作

# 创建生成 kubelet 配置脚本
$ vim kubelet.sh
#!/bin/bash

DNS_SERVER_IP=${1:-"10.10.0.2"}
HOSTNAME=${2:-"`hostname`"}
CLUETERDOMAIN=${3:-"cluster.local"}

cat <<EOF >/opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=true \\
--v=2 \\
--hostname-override=${HOSTNAME} \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--network-plugin=cni \\
--cni-conf-dir=/etc/cni/net.d \\
--cni-bin-dir=/opt/cni/bin \\
--pod-infra-container-image=yangpeng2468/google_containers-pause-amd64:3.2"
EOF

cat <<EOF >/opt/kubernetes/cfg/kubelet-config.yml
kind: KubeletConfiguration # 使用对象
apiVersion: kubelet.config.k8s.io/v1beta1 # api版本
address: 0.0.0.0 # 监听地址
port: 10250 # 当前kubelet的端口
readOnlyPort: 10255 # kubelet暴露的端口
cgroupDriver: cgroupfs # 驱动，要于docker info显示的驱动一致
clusterDNS:
  - ${DNS_SERVER_IP}
clusterDomain: ${CLUETERDOMAIN}  # 集群域
failSwapOn: false # 关闭swap

# 身份验证
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem

# 授权
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s

# Node 资源保留
evictionHard:
  imagefs.available: 15%
  memory.available: 1G
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s

# 镜像删除策略
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s

# 旋转证书
rotateCertificates: true # 旋转kubelet client 证书
featureGates:
  RotateKubeletServerCertificate: true
  RotateKubeletClientCertificate: true

maxOpenFiles: 1000000
maxPods: 110
EOF

cat <<EOF >/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet

# 创建生成 kube-proxy 配置脚本
$ vim proxy.sh
#!/bin/bash

HOSTNAME=${1:-"`hostname`"}

cat <<EOF >/opt/kubernetes/cfg/kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=true \\
--v=2 \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

cat <<EOF >/opt/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0 # 监听地址
metricsBindAddress: 0.0.0.0:10249 # 监控指标地址,监控获取相关信息 就从这里获取
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig # 读取配置文件
hostnameOverride: ${HOSTNAME} # 注册到k8s的节点名称唯一
clusterCIDR: 10.20.0.0/16 # Pod IP范围
mode: iptables # 使用iptables模式

# 使用 ipvs 模式
#mode: ipvs # ipvs 模式
#ipvs:
#  scheduler: "rr"
#iptables:
#  masqueradeAll: true
EOF

cat <<EOF >/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy

# 生成 node 配置文件
$ ./kubelet.sh 10.10.0.2 k8s-master1 cluster.local
$ ./proxy.sh k8s-master1

# 查看服务是否启动
$ netstat -ntpl | egrep "kubelet|kube-proxy"

# copy kubelet.sh proxy.sh 脚本到 k8s-master2 k8s-master3 机器上
$ scp kubelet.sh  proxy.sh  root@k8s-master2:/data/k8s-node
$ scp kubelet.sh  proxy.sh  root@k8s-master3:/data/k8s-node

登陆到 k8s-master2 操作

$ cd /data/k8s-node

# 生成 node 配置文件
$ ./kubelet.sh 10.10.0.2 k8s-master2 cluster.local
$ ./proxy.sh k8s-master2

# 查看服务是否启动
$ netstat -ntpl | egrep "kubelet|kube-proxy"

登陆到 k8s-master3 操作

$ cd /data/k8s-node

# 生成 node 配置文件
$ ./kubelet.sh 10.10.0.2 k8s-master3 cluster.local
$ ./proxy.sh k8s-master3

# 查看服务是否启动
$ netstat -ntpl | egrep "kubelet|kube-proxy"

# 随便登陆一台master机器查看node节点是否添加成功
$ kubectl get node

NAME       STATUS   ROLES    AGE    VERSION
k8s-master1   NoReady    <none>   4d4h   v1.18.2
k8s-master2   NoReady    <none>   4d4h   v1.18.2
k8s-master3   NoReady    <none>   4d4h   v1.18.2

上面 Node 节点处理 NoReady 状态，是因为目前还没有安装网络组件，下文安装网络组件。

$ vim ~/yaml/apiserver-to-kubelet-rbac.yml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-api-admin
subjects:
- kind: User
  name: kubernetes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io
# 应用
$ kubectl apply -f ~/yaml/apiserver-to-kubelet-rbac.yml

3.9 安装calico网络，使用IPIP模式

登陆到 k8s-master1 操作

下载 Calico Version v3.14.0 Yaml 文件

# 存放etcd yaml文件
$ mkdir -p ~/yaml/calico

$ cd ~/yaml/calico

# 注意：下面是基于自建etcd做为存储的配置文件
$ curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -O

calico-etcd.yaml 需要修改如下配置：

Secret 配置修改

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  etcd-key: (cat /opt/kubernetes/ssl/server-key.pem | base64 -w 0) # 将输出结果填写在这里
  etcd-cert: (cat /opt/kubernetes/ssl/server.pem | base64 -w 0) # 将输出结果填写在这里
  etcd-ca: (cat /opt/kubernetes/ssl/ca.pem | base64 -w 0) # 将输出结果填写在这里

ConfigMap 配置修改

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  etcd_endpoints: "https://192.168.0.216:2379,https://192.168.0.217:2379,https://192.168.0.218:2379"
  etcd_ca: "/calico-secrets/etcd-ca"
  etcd_cert: "/calico-secrets/etcd-cert"
  etcd_key: "/calico-secrets/etcd-key"

关于ConfigMap部分主要参数如下：

etcd_endpoints：Calico使用etcd来保存网络拓扑和状态，该参数指定etcd的地址，可以使用K8S Master所用的etcd，也可以另外搭建。
calico_backend：Calico的后端，默认为bird。
cni_network_config：符合CNI规范的网络配置，其中type=calico表示，Kubelet 从 CNI_PATH (默认为/opt/cni/bin)目录找calico的可执行文件，用于容器IP地址的分配。
etcd 如果配置TLS安全认证，则还需要指定相应的ca、cert、key等文件

修改 Pods 使用的 IP 网段，默认使用 192.168.0.0/16 网段

 - name: CALICO_IPV4POOL_CIDR
              value: "10.20.0.0/16"

配置网卡自动发现规则

在 DaemonSet calico-node env 中添加网卡发现规则

 # 定义ipv4自动发现网卡规则
            - name: IP_AUTODETECTION_METHOD
              value: "interface=eth.*"
            # 定义ipv6自动发现网卡规则
            - name: IP6_AUTODETECTION_METHOD
              value: "interface=eth.*"

Calico 模式设置

  # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"

Calico 有两种网络模式：BGP 和 IPIP

使用 IPIP 模式时，设置 CALICO_IPV4POOL_IPIP="always"，IPIP 是一种将各Node的路由之间做一个tunnel，再把两个网络连接起来的模式，启用IPIP模式时，Calico将在各Node上创建一个名为 tunl0 的虚拟网络接口。
使用 BGP 模式时，设置 CALICO_IPV4POOL_IPIP="off"

错误解决方法

错误：[ERROR][8] startup/startup.go 146: failed to query kubeadm's config map error=Get https://10.10.0.1:443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=2s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

原因：Node工作节点连接不到 apiserver 地址，检查一下calico配置文件，要把apiserver的IP和端口配置上，如果不配置的话，calico默认将设置默认的calico网段和443端口。字段名：KUBERNETES_SERVICE_HOST、KUBERNETES_SERVICE_PORT、KUBERNETES_SERVICE_PORT_HTTPS。

解决方法：

在 DaemonSet calico-node env 中添加环境变量

 - name: KUBERNETES_SERVICE_HOST
              value: "lb.ypvip.com.cn"
            - name: KUBERNETES_SERVICE_PORT
              value: "6443"
            - name: KUBERNETES_SERVICE_PORT_HTTPS
              value: "6443"

修改完 calico-etcd.yaml 后，执行部署

# 部署
$ kubectl apply -f calico-etcd.yaml

# 查看 calico pods
$ kubectl  get pods -n kube-system  | grep calico

# 查看 node 是否正常，现在 node 服务正常了
$ kubectl get node

NAME       STATUS   ROLES    AGE    VERSION
k8s-master1   Ready    <none>   4d4h   v1.18.2
k8s-master2   Ready    <none>   4d4h   v1.18.2
k8s-master3   Ready    <none>   4d4h   v1.18.2

3.10 集群CoreDNS部署

登陆到 k8s-master1 操作

deploy.sh 是一个便捷的脚本，用于生成coredns yaml 配置。

# 安装依赖 jq 命令
$ yum  install  jq -y

$ cd ~/yaml
$ mkdir coredns
$ cd coredns

# 下载 CoreDNS 项目
$ git clone https://github.com/coredns/deployment.git
$ cd coredns/deployment/kubernetes

默认情况下 CLUSTER_DNS_IP 是自动获取kube-dns的集群ip的，但是由于没有部署kube-dns所以只能手动指定一个集群ip。

## vim deployment/kubernetes/deploy.sh

if [[ -z $CLUSTER_DNS_IP ]]; then
   # Default IP to kube-dns IP
   # CLUSTER_DNS_IP=$(kubectl get service --namespace kube-system kube-dns -o jsonpath="{.spec.clusterIP}")
    CLUSTER_DNS_IP=10.10.0.2

# 查看执行效果，并未开始部署

$ ./deploy.sh

# 执行部署
$ ./deploy.sh | kubectl apply -f -

# 查看 Coredns
$ kubectl get svc,pods -n kube-system| grep coredns

测试 Coredns 解析

# 创建一个 busybox Pod
$ vim busybox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.4
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

# 部署
$ kubectl apply -f busybox.yaml

# 测试解析，下面是解析正常
$ kubectl exec -i busybox -n default nslookup kubernetes

Server:    10.10.0.2
Address 1: 10.10.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.10.0.1 kubernetes.default.svc.cluster.local

3.11 部署集群监控服务 Metrics Server

登陆到 k8s-master1 操作

$ cd ~/yaml

# 拉取 v0.3.6 版本
$ git clone https://github.com/kubernetes-sigs/metrics-server.git -b v0.3.6
$ cd metrics-server/deploy/1.8+

只修改 metrics-server-deployment.yaml 配置文件

# 下面是修改前后比较差异
$ git diff metrics-server-deployment.yaml

diff --git a/deploy/1.8+/metrics-server-deployment.yaml b/deploy/1.8+/metrics-server-deployment.yaml
index 2393e75..2139e4a 100644
--- a/deploy/1.8+/metrics-server-deployment.yaml
+++ b/deploy/1.8+/metrics-server-deployment.yaml
@@ -29,8 +29,19 @@ spec:
         emptyDir: {}
       containers:
       - name: metrics-server
-        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
-        imagePullPolicy: Always
+        image: yangpeng2468/metrics-server-amd64:v0.3.6
+        imagePullPolicy: IfNotPresent
+        resources:
+          limits:
+            cpu: 400m
+            memory: 1024Mi
+          requests:
+            cpu: 50m
+            memory: 50Mi
+        command:
+        - /metrics-server
+        - --kubelet-insecure-tls
+        - --kubelet-preferred-address-types=InternalIP
         volumeMounts:
         - name: tmp-dir
           mountPath: /tmp

# 部署
$ kubectl apply -f .

# 验证
$ kubectl top node

NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master1   72m          7%     1002Mi          53%
k8s-master2   121m         3%     1852Mi          12%
k8s-master3   300m         3%     1852Mi          20%

# 内存单位 Mi=1024*1024字节  M=1000*1000字节
# CPU单位 1核=1000m 即 250m=1/4核

3.12 部署 Kubernetes Dashboard

Kubernetes Dashboard 部署，请参考 K8S Dashboard 2.0 部署并使用 Ingress-Nginx 提供访问入口文章。

3.13 部署Nginx负载均衡器

kube-apiserver高可用架构图：

安装软件包（主/备）

yum install epel-release -y
yum install nginx keepalived -y

Nginx配置文件（主/备一样）

cat > /etc/nginx/nginx.conf << "EOF"
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.31.71:6443;   # Master1 APISERVER IP:PORT
       server 192.168.31.74:6443;   # Master2 APISERVER IP:PORT
    }
    
    server {
       listen 6443;
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF

keepalived配置文件（Nginx Master）

cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}
vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 100    # 优先级，备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 虚拟IP
    virtual_ipaddress {
        192.168.31.88/24
    }
    track_script {
        check_nginx
    }
}
EOF

检查nginx状态脚本：

cat > /etc/keepalived/check_nginx.sh  << "EOF"
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh

keepalived配置文件（Nginx Backup）

cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_BACKUP
}
vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.31.88/24
    }
    track_script {
        check_nginx
    }
}
EOF

访问负载均衡器测试

kill掉一台master，使用curl查看K8s版本测试，使用VIP访问

curl -k https://192.168.31.88:6443/version
{
  "major": "1",
  "minor": "18",
  "gitVersion": "v1.18.3",
  "gitCommit": "2e7996e3e2712684bc73f0dec0200d64eec7fe40",
  "gitTreeState": "clean",
  "buildDate": "2020-05-20T12:43:34Z",
  "goVersion": "go1.13.9",
  "compiler": "gc",
  "platform": "linux/amd64"
}

修改所有Worker Node连接LB VIP

虽然增加了Master2和负载均衡器，但是是从单Master架构扩容的，也就是说目前所有的Node组件连接都还是Master1，如果不改为连接VIP走负载均衡器，那么Master还是单点故障。

因此接下来就是要改所有Node组件配置文件，由原来192.168.31.71修改为192.168.31.88（VIP）：

在上述所有Worker Node执行：

sed -i 's#192.168.31.71:6443#192.168.31.88:6443#' /opt/kubernetes/cfg/*
systemctl restart kubelet
systemctl restart kube-proxy

3.14 haproxy负载均衡方式

所有Master节点通过yum安装HAProxy和KeepAlived：

yum install keepalived haproxy -y

所有Master节点配置HAProxy（详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同）：

# mkdir /etc/haproxy
[root@k8s-master01 etc]# vim /etc/haproxy/haproxy.cfg 
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.0.201:6443  check
  server k8s-master02	192.168.0.202:6443  check
  server k8s-master03	192.168.0.203:6443  check

所有Master节点配置KeepAlived，配置不一样，注意区分
注意每个节点的IP和网卡（interface参数）
Master01节点的配置：

mkdir /etc/keepalived

[root@k8s-master01 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens192
    mcast_src_ip 192.168.0.201
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.0.236
    }
    track_script {
       chk_apiserver
    }
}

Master02节点的配置：

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
   interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens192
    mcast_src_ip 192.168.0.202
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.0.236
    }
    track_script {
       chk_apiserver
    }
}

Master03节点的配置：

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
 interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens192
    mcast_src_ip 192.168.0.203
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.0.236
    }
    track_script {
       chk_apiserver
    }
}

配置KeepAlived健康检查文件：

# cat /etc/keepalived/check_apiserver.sh 
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi


chmod +x /etc/keepalived/check_apiserver.sh

启动haproxy和keepalived
# systemctl daemon-reload
# systemctl enable --now haproxy
# systemctl enable --now keepalived

https://mp.weixin.qq.com/s/Hyf0OD7KPeoBJw8gxTQsYA Kubernetes v1.18.2 二进制高可用部署(修正版)

https://mp.weixin.qq.com/s/az3OgOOTi17YpH0pmVDM-g 高可用安装K8s集群1.20.x

https://mp.weixin.qq.com/s/VYtyTU9_Dw9M5oHtvRfseA 部署一套完整的Kubernetes高可用集群（上）

https://mp.weixin.qq.com/s/F9BC6GALHiWBK5dmUnqepA 部署一套完整的Kubernetes高可用集群（下）

posted @ 2020-06-28 09:08 凡人半睁眼阅读(550) 评论(0) 收藏举报

刷新页面返回顶部

海棠未雨，梨花先雪，一半春休

想看山海，早也去，晚也去，一个人也去

二进制 k8s集群（docker）

一、环境

节点信息

服务版本与K8S集群说明

Service 和 Pods Ip 段划分

Service 和 Pods Ip 段划分

二、环境初始化

三、Kubernetes 部署

部署顺序

3.1 自签TLS证书

3.3 创建 metrics-server 证书

3.4 获取K8S二进制包

3.5 创建Node节点kubeconfig文件

3.6 配置Master组件并运行

3.7 配置kubelet证书自动续期和创建Node授权用户

3.8 配置Node组件并运行

3.9 安装calico网络，使用IPIP模式

3.10 集群CoreDNS部署

3.11 部署集群监控服务 Metrics Server

3.12 部署 Kubernetes Dashboard

3.13 部署Nginx负载均衡器

Nginx配置文件（主/备一样）

3.14 haproxy负载均衡方式

公告

海棠未雨，梨花先雪，一半春休

想看山海，早也去，晚也去 ，一个人也去

二进制 k8s集群（docker）

一、环境

节点信息

服务版本与K8S集群说明

Service 和 Pods Ip 段划分

Service 和 Pods Ip 段划分

二、环境初始化

三、Kubernetes 部署

部署顺序

3.1 自签TLS证书

3.3 创建 metrics-server 证书

3.4 获取K8S二进制包

3.5 创建Node节点kubeconfig文件

3.6 配置Master组件并运行

3.7 配置kubelet证书自动续期和创建Node授权用户

3.8 配置Node组件并运行

3.9 安装calico网络，使用IPIP模式

3.10 集群CoreDNS部署

3.11 部署集群监控服务 Metrics Server

3.12 部署 Kubernetes Dashboard

3.13 部署Nginx负载均衡器

Nginx配置文件（主/备一样）

3.14 haproxy负载均衡方式

公告

想看山海，早也去，晚也去，一个人也去