学习K8S之路.3--- K8S集群安装部署(二)
上节在运算节点(192.168.6.94和192.168.6.95)上安装部署完成kube-apiserver，本节，继续往下部署
一：在代理节点(192.168.6.92和192.168.6.93)上安装nginx+keepalived,对apiserver进行四层反向代理
1.1：安装Nginx
1.1.1：下载，编译，安装
-6-92 ~]# useradd -s /sbin/nologin -M www -6-92 ~]# tar zxf nginx-1.16.1.tar.gz -6-92 ~]# cd nginx-1.16.1 -6-92 nginx-1.16.1]# ./configure --user=www --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-pcre --with-stream -6-92 nginx-1.16.1]# make && make install
1.1.2：修改nginx.conf配置文件
http 和 stream是两个不同的模块，所以不能写在一些 在http下面新增stream的配置信息 stream { include /usr/local/nginx/conf/ports/*.conf; }
1.1.3：配置四层反向代理
-6-92 conf]# mkdir /usr/local/nginx/conf/ports/ -6-92 conf]# vi /usr/local/nginx/conf/ports/apiserver.conf upstream kube-apiserver { server 192.168.6.94:6443 max_fails=3 fail_timeout=30s; server 192.168.6.95:6443 max_fails=3 fail_timeout=30s; } server { listen 7443; proxy_connect_timeout 2s; proxy_timeout 900s; proxy_pass kube-apiserver; }
1.1.4：检查配置文件，并启动
/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx
1.2：安装Keepalived服务
1.2.1：安装
-6-92 ~]# yum install keepalived -y
1.2.2：编辑keepalived检测脚本
-6-92 ~]# vi /etc/keepalived/check_port.sh #!/bin/bash CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l` if [ $PORT_PROCESS -eq 0 ];then systemctl stop keepalived exit 1 fi else echo "Check Port Cant Be Empty!" fi
1.2.3：添加执行权限
-6-92 ~]# chmod +x /etc/keepalived/check_port.sh -6-92 ~]# ll /etc/keepalived/check_port.sh -rwxr-xr-x 1 root root 281 5月 3 19:14 /etc/keepalived/check_port.sh
1.2.4：配置keepalived配置文件
keepalived主配置文件：
-6-92 ~]# vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id 192.168.6.92 } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 7443" interval 2 } vrrp_instance VI_1 { state BACKUP interface eth0 //注意网卡名称 virtual_router_id 251 priority 100 advert_int 1 nopreempt authentication { auth_type PASS auth_pass 11111111 } track_script { chk_nginx } virtual_ipaddress { 192.168.6.89 } }
//nopreempt：非抢占式
keepalived从配置文件
-6-93 ~]# vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id 192.168.6.93 } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 7443" interval 2 } vrrp_instance VI_1 { state BACKUP interface eth0 //注意网卡名称 virtual_router_id 251 priority 90 nopreempt
authentication { auth_type PASS auth_pass 11111111 } track_script { chk_nginx } virtual_ipaddress { 192.168.6.89 } }
1.2.5：启动keepalived，并验证keepalived
-6-92 ~]# systemctl start keepalived -6-92 ~]# systemctl enable keepalived -6-92 ~]# ip addr
注意：keepalived 配置 nopreempt，意为非抢占式
原因：如果抢占式，假如生产网络抖动原因，check_port脚本探测不到，VIP有可能会动触发报警，VIP漂移在生产上属于重大生产事故，是要写故障报告的，是无法忍受的
二：部署controller-manager
| 主机名 | 角色 | IP地址 |
| k8s-6-94.host.com | controller-manager | 192.168.6.94 |
| k8s-6-95.host.com | controller-manager | 192.168.6.95 |
在运算节点(192.168.6.94和192.168.6.95)上部署controller-manager
以192.168.6.94部署为例：
1：创建启动脚本
-6-94 ~]# vi /opt/kubernetes/server/bin/kube-controller-manager.sh #!/bin/sh ./kube-controller-manager \ --cluster-cidr 172.6.0.0/16 \ --leader-elect true \ --log-dir /data/logs/kubernetes/kube-controller-manager \ --master http://127.0.0.1:8080 \ --service-account-private-key-file ./certs/ca-key.pem \ --service-cluster-ip-range 10.100.0.0/16 \ --root-ca-file ./certs/ca.pem \ --v 2 // 注解： --cluster-cidr 172.6.0.0/16：集群的容器的网段
2：授权文件权限，创建目录
-6-94 ~]# chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh -6-94 ~]# mkdir -p /data/logs/kubernetes/kube-controller-manager
3：创建supervisor配置
-6-94 ~]# vi /etc/supervisord.d/kube-conntroller-manager.ini [program:kube-controller-manager-6-94] command=/opt/kubernetes/server/bin/kube-controller-manager.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
4：启动服务并检查
-6-94 ~]# supervisorctl update -6-94 ~]# supervisorctl status
5：部署集群其他服务器
# 不同的的地方 /etc/supervisord.d/kube-conntroller-manager.ini [program:kube-controller-manager-6-94]
三：部署kube-scheduler
| 主机名 | 角色 | IP地址 |
| k8s-6-94.host.com | kube-scheduler | 192.168.6.94 |
| k8s-6-95.host.com | kube-scheduler | 192.168.6.95 |
在运算节点(192.168.6.94和192.168.6.95)上部署controller-manager
以192.168.6.94部署为例：
1：创建启动脚本
-6-94 ~]# vi /opt/kubernetes/server/bin/kube-scheduler.sh #!/bin/sh ./kube-scheduler \ --leader-elect \ --log-dir /data/logs/kubernetes/kube-scheduler \ --master http://127.0.0.1:8080 \ --v 2
2：授权文件权限，创建目录
-6-94 ~]# chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh -6-94 ~]# mkdir -p /data/logs/kubernetes/kube-scheduler
3：创建supervisor配置
-6-94 ~]# vi /etc/supervisord.d/kube-scheduler.ini [program:kube-scheduler-6-94] command=/opt/kubernetes/server/bin/kube-scheduler.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false
4：启动服务并检查
-6-94 ~]# supervisorctl update kube-scheduler-6-94: added process group -6-94 ~]# supervisorctl status
5：部署集群其他服务器
# 不同的地方 /etc/supervisord.d/kube-scheduler.ini [program:kube-scheduler-6-94]
四：检查master节点
1：建立kubectl软链接
-6-94 ~]# ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
2：检查master节点集群状态
-6-94 ~]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} -6-94 ~]# kubectl get namespaces NAME STATUS AGE default Active 30h kube-node-lease Active 30h kube-public Active 30h kube-system Active 30h
至此，主控制节点已经部署完成
五：接下来部署node节点
| 主机名称 | 部署服务 | IP地址 |
| k8s-6-94.host.com | kubelet，kube-proxy | 192.168.6.94 |
| k8s-6-95.host.com | kubelet，kube-proxy | 192.168.6.95 |
5.1：在运算节点(192.168.6.94和192.168.6.95)上部署kubelet
5.1.1：在运维主机(192.168.6.96)上创建证书
5.1.1.1：创建生成证书签名请求（csr）的json配置文件
-6-96 ~]# cd /opt/certs/ -6-96 certs]# vi kubelet-csr.json { "CN": "k8s-kubelet", "hosts": [ "127.0.0.1", "192.168.6.89", "192.168.6.92", "192.168.6.93", "192.168.6.94", "192.168.6.95", "192.168.6.96", "192.168.6.97" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ] }
5.1.1.2：生成kubelet证书和私钥
-6-96 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet 2020/05/04 16:13:08 [INFO] generate received request 2020/05/04 16:13:08 [INFO] received CSR 2020/05/04 16:13:08 [INFO] generating key: rsa-2048 2020/05/04 16:13:08 [INFO] encoded CSR 2020/05/04 16:13:08 [INFO] signed certificate with serial number 15973371575590796904243905182545744769790158254 2020/05/04 16:13:08 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").
5.1.1.3：检查生成的证书和私钥
-6-96 certs]# ll | grep kubelet -rw-r--r-- 1 root root 1098 5月 4 16:13 kubelet.csr -rw-r--r-- 1 root root 439 5月 4 16:12 kubelet-csr.json -rw------- 1 root root 1675 5月 4 16:13 kubelet-key.pem -rw-r--r-- 1 root root 1448 5月 4 16:13 kubelet.pem
5.1.2：拷贝证书到各运算节点
以192.168.6.94部署为例：
-6-94 ~]# cd /opt/kubernetes/server/bin/certs -6-94 certs]# scp 192.168.6.96:/opt/certs/kubelet.pem . -6-94 certs]# scp 192.168.6.96:/opt/certs/kubelet-key.pem . -6-94 certs]# ll | grep kubelet -rw------- 1 root root 1675 5月 4 16:17 kubelet-key.pem -rw-r--r-- 1 root root 1448 5月 4 16:17 kubelet.pem
5.1.3：创建配置
以192.168.6.94部署为例：注意：必须在conf目录下，否则报错
5.1.3.1：set-cluster
-6-94 ~]# cd /opt/kubernetes/server/bin/conf -6-94 conf]# kubectl config set-cluster myk8s \ --certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \ --embed-certs=true \ --server=https://192.168.6.89:7443 \ --kubeconfig=kubelet.kubeconfig 返回结果： Cluster "myk8s" set.
5.1.3.2：set-credentials
-6-94 conf]# kubectl config set-credentials k8s-node \ --client-certificate=/opt/kubernetes/server/bin/certs/client.pem \ --client-key=/opt/kubernetes/server/bin/certs/client-key.pem \ --embed-certs=true \ --kubeconfig=kubelet.kubeconfig 返回结果： User "k8s-node" set.
5.1.3.3：set-context
-6-94 conf]# kubectl config set-context myk8s-context \ --cluster=myk8s \ --user=k8s-node \ --kubeconfig=kubelet.kubeconfig 返回结果： Context "myk8s-context" created.
5.1.3.4：use-context
-6-94 conf]# kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig 返回结果 Switched to context "myk8s-context".
5.1.3.5：查看生成的kubelet.kubeconfig
-6-94 conf]# ll -rw-r--r-- 1 root root 2223 5月 3 22:38 audit.yaml -rw------- 1 root root 6186 5月 4 16:24 kubelet.kubeconfig
5.1.4：创建k8s-node.yaml
注意：必须在conf目录下，否则报错
5.1.4.1：创建配置文件
-6-94 ~]# cd /opt/kubernetes/server/bin/conf -6-94 conf]# vi k8s-node.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: k8s-node roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: k8s-node
5.1.4.2：应用资源配置
-6-94 conf]# kubectl create -f k8s-node.yaml //-f 指定配置文件 clusterrolebinding.rbac.authorization.k8s.io/k8s-node created //创建角色后会存到etcd里
5.1.4.3：查看集群角色和角色属性
[root@k8s-6-94 conf]# kubectl get clusterrolebinding k8s-node NAME AGE k8s-node 33s [root@k8s-6-94 conf]# kubectl get clusterrolebinding k8s-node -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding //定义了一个集群绑定资源，k8s一切皆资源 metadata: creationTimestamp: "2020-05-04T08:31:33Z" name: k8s-node //资源的名称为k8s-node resourceVersion: "107039" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node uid: 2206c4c7-bef3-40ff-84f8-50eb298470da roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node //给k8s-node集群用户绑定了一个集群角色，叫system：node，意思是让下面k8s-node用户具备这个集群里运算节点的权限 subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: k8s-node
5.1.5：拷贝kubelet.kubeconfig 到其他运算节点(192.168.6.95)上的conf目录下
-6-95 ]# cd /opt/kubernetes/server/bin/conf -6-95 conf]# scp 192.168.6.94:/opt/kubernetes/server/bin/conf/kubelet.kubeconfig . //注：kubelet.kubeconfig 此文件拷贝至集群其他服务器，可以省略5.1.3(创建配置)中的四步和5.1.4(角色绑定)，其实5.1.3中的四步，就是生成kubelet.kubeconfig文件
5.1.6：准备pause基础镜像，在运维主机(192.168.6.96)上操作
为什么需要这个pause基础镜像？
原因：需要用一个pause基础镜像把这台机器的pod拉起来，因为kubelet是干活的节点，它帮我们调度docker引擎，边车模式，让kebelet控制一个小镜像，先于我们的业务容器起来，让它帮我们业务容器去设置：UTC、NET、IPC，让它先把命名空间占上，业务容易还没起来的时候，pod的ip已经分配出来了。
5.1.6.1：下载pause镜像
-6-96 ~]# docker pull kubernetes/pause
5.1.6.2：给pause镜像打tag
-6-96 certs]# docker images -a -6-96 certs]# docker tag f9d5de079539 harbor.auth.com/public/pause:latest -6-96 certs]# dockerimages -a
5.1.6.3：上传到私有仓库harbor上
-6-96 certs]# docker push harbor.auth.com/public/pause:latest
5.1.7：创建kubelet启动脚本
-6-94 ~]# cd /opt/kubernetes/server/bin -6-94 bin]# vi /opt/kubernetes/server/bin/kubelet.sh #!/bin/sh ./kubelet \ --anonymous-auth=false \ --cgroup-driver systemd \ --cluster-dns 10.100.0.2 \ --cluster-domain cluster.local \ --runtime-cgroups=/systemd/system.slice \ --kubelet-cgroups=/systemd/system.slice \ --fail-swap-on="false" \ --client-ca-file ./certs/ca.pem \ --tls-cert-file ./certs/kubelet.pem \ --tls-private-key-file ./certs/kubelet-key.pem \ --hostname-override k8s-6-94.host.com \ --image-gc-high-threshold 20 \ --image-gc-low-threshold 10 \ --kubeconfig ./conf/kubelet.kubeconfig \ --log-dir /data/logs/kubernetes/kube-kubelet \ --pod-infra-container-image harbor.auth.com/public/pause:latest \ --root-dir /data/kubelet
5.1.8：授权，创建目录
-6-94 bin]# chmod +x /opt/kubernetes/server/bin/kubelet.sh -6-94 bin]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
5.1.9：创建supervisor配置
-6-94 bin]# vi /etc/supervisord.d/kube-kubelet.ini [program:kube-kubelet-6-94] command=/opt/kubernetes/server/bin/kubelet.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
5.1.10：启动服务并检查
-6-94 bin]# supervisorctl update kube-kubelet-6-94: added process group -6-94 bin]# supervisorctl status
5.1.11：部署集群其他服务器
# 不同的地方 /opt/kubernetes/server/bin/kubelet.sh --hostname-override ########## /etc/supervisord.d/kube-kubelet.ini [program:kube-kubelet-6-94]
5.1.12：检查所有节点并给节点打上标签
[root@k8s-6-94 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-6-94.host.com Ready <none> 66m v1.15.4 k8s-6-95.host.com Ready <none> 58m v1.15.4 // 角色是none，打个标签，标签是特色管理功能之一 [root@k8s-6-94 ~]# kubectl label node k8s-6-94.host.com node-role.kubernetes.io/master= node/k8s-6-94.host.com labeled [root@k8s-6-94 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-6-94.host.com Ready master 66m v1.15.4 k8s-6-95.host.com Ready <none> 59m v1.15.4 [root@k8s-6-94 ~]# kubectl label node k8s-6-94.host.com node-role.kubernetes.io/node= node/k8s-6-94.host.com labeled [root@k8s-6-94 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-6-94.host.com Ready master,node 67m v1.15.4 k8s-6-95.host.com Ready <none> 59m v1.15.4
5.2：在运算节点(192.168.6.94和192.168.6.95)上部署kube-proxy
5.2.1：在运维主机(192.168.6.96)上创建证书
5.2.1.1：创建生成证书csr的json配置文件
-6-96 ~]# cd /opt/certs/ -6-96 certs]# vi kube-proxy-csr.json { "CN": "system:kube-proxy", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ] }
5.2.1.2：生成kube-proxy证书文件
-6-96 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client 2020/05/04 17:56:35 [INFO] generate received request 2020/05/04 17:56:35 [INFO] received CSR 2020/05/04 17:56:35 [INFO] generating key: rsa-2048 2020/05/04 17:56:35 [INFO] encoded CSR 2020/05/04 17:56:35 [INFO] signed certificate with serial number 151736134429961131161009868543598459373320285897 2020/05/04 17:56:35 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").
5.2.1.3：检查生成的证书和私钥
-6-96 certs]# ll | grep kube-proxy-client -rw-r--r-- 1 root root 1005 5月 4 17:56 kube-proxy-client.csr -rw------- 1 root root 1679 5月 4 17:56 kube-proxy-client-key.pem -rw-r--r-- 1 root root 1375 5月 4 17:56 kube-proxy-client.pem
5.2.2：拷贝证书文件至各节点
以192.168.6.94部署为例： -6-94 ~]# cd /opt/kubernetes/server/bin/certs/ -6-94 certs]# scp 192.168.6.96:/opt/certs/kube-proxy-client.pem . -6-94 certs]# scp 192.168.6.96:/opt/certs/kube-proxy-client-key.pem .
5.2.3：创建配置，注意必须在conf目录下，否则报错
5.2.3.1：set-cluster
-6-94 ~]# cd /opt/kubernetes/server/bin/conf -6-94 conf]# kubectl config set-cluster myk8s \ --certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \ --embed-certs=true \ --server=https://192.168.6.89:7443 \ --kubeconfig=kube-proxy.kubeconfig 返回结果 Cluster "myk8s" set.
5.2.3.2：set-credentials
-6-94 conf]# kubectl config set-credentials kube-proxy \ --client-certificate=/opt/kubernetes/server/bin/certs/kube-proxy-client.pem \ --client-key=/opt/kubernetes/server/bin/certs/kube-proxy-client-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig 返回结果： User "kube-proxy" set.
5.2.3.3：set-context
-6-94 conf]# kubectl config set-context myk8s-context \ --cluster=myk8s \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig 返回结果： Context "myk8s-context" created.
5.2.3.4：use-context
-6-94 conf]# kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig 返回结果： Switched to context "myk8s-context".
5.2.4：拷贝kube-proxy.kubeconfig 到其他运算节点(192.168.6.95)上的conf目录下
-6-95 ~]# cd /opt/kubernetes/server/bin/conf -6-95 conf]# scp 192.168.6.94:/opt/kubernetes/server/bin/conf/kube-proxy.kubeconfig . //proxy.kubeconfig 此文件拷贝至集群其他服务器，可以省略5.2.3(创建配置),其实5.2.3中的四步就是生成proxy.kubeconfig文件
5.2.5：加载ipvs模块
1：查看ipvs模块 [root@k8s-6-94 ~]# lsmod |grep ip_vs 2：编辑开启ipvs模块脚本 [root@k8s-6-94 ~]# vi /root/ipvs.sh #!/bin/bash ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs" for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*") do /sbin/modinfo -F filename $i &>/dev/null if [ $? -eq 0 ];then /sbin/modprobe $i fi done 3：添加执行权限，开启ipvs模块，并验证 [root@k8s-6-94 ~]# chmod +x /root/ipvs.sh [root@k8s-6-94 ~]# sh /root/ipvs.sh [root@k8s-6-94 ~]# lsmod |grep ip_vs ip_vs_wrr 12697 0 ip_vs_wlc 12519 0 ip_vs_sh 12688 0 ip_vs_sed 12519 0 ip_vs_rr 12600 0 ip_vs_pe_sip 12697 0 ..........
5.2.6：创建启动脚本
-6-94 ~]# vi /opt/kubernetes/server/bin/kube-proxy.sh #!/bin/sh ./kube-proxy \ --cluster-cidr 172.6.0.0/16 \ --hostname-override k8s-6-94.host.com \ --proxy-mode=ipvs \ --ipvs-scheduler=nq \ --kubeconfig ./conf/kube-proxy.kubeconfig
5.2.7：检查配置，权限，创建日志目录
[root@k8s-6-94 ~]# ls -l /opt/kubernetes/server/bin/conf/|grep kube-proxy -rw------- 1 root root 6210 5月 4 18:06 kube-proxy.kubeconfig [root@k8s-6-94 ~]# chmod +x /opt/kubernetes/server/bin/kube-proxy.sh [root@k8s-6-94 ~]# mkdir -p /data/logs/kubernetes/kube-proxy
5.2.8：创建supervisor配置
[root@k8s-6-94 ~]# vi /etc/supervisord.d/kube-proxy.ini [program:kube-proxy-6-94] command=/opt/kubernetes/server/bin/kube-proxy.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=root ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log ; stderr log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=4 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
5.2.9：启动服务并检查
[root@k8s-6-94 ~]# supervisorctl update kube-proxy-6-94: added process group [root@k8s-6-94 ~]# supervisorctl status etcd-server-6-94 RUNNING pid 12780, uptime 19:32:21 kube-apiserver-6-94 RUNNING pid 12777, uptime 19:32:21 kube-controller-manager-6-94 RUNNING pid 12778, uptime 19:32:21 kube-kubelet-6-94 RUNNING pid 14293, uptime 1:34:45 kube-proxy-6-94 RUNNING pid 2940, uptime 0:03:08 kube-scheduler-6-94 RUNNING pid 12779, uptime 19:32:21 [root@k8s-6-94 ~]# yum install ipvsadm -y [root@k8s-6-94 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.100.0.1:443 nq -> 192.168.6.94:6443 Masq 1 0 0 -> 192.168.6.95:6443 Masq 1 0 0 [root@k8s-6-94 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 2d2h
5.2.10：部署集群其他服务器
# 不同的地方 /opt/kubernetes/server/bin/kube-proxy.sh --hostname-override ########## /etc/supervisord.d/kube-proxy.ini [program:kube-proxy-6-94]
六：验证kubernetes集群
1：在任意一个运算节点(192.168.6.94)，创建一个资源配置清单
//私有仓库需要上传nginx 1.7.9版本
[root@k8s-6-94 ~]# vi nginx-ds.yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nginx-ds spec: template: metadata: labels: app: nginx-ds spec: containers: - name: my-nginx image: harbor.auth.com/public/nginx:1.7.9 ports: - containerPort: 80
2：在运算节点(192.168.6.94)上应用资源配置，并检查
1：应用资源配置 [root@k8s-6-94 ~]# kubectl create -f /root/nginx-ds.yaml daemonset.extensions/nginx-ds created 2：检查 [root@k8s-6-94 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-ds-h5gw5 1/1 Running 0 24s nginx-ds-mxjcd 1/1 Running 0 24s # 扩展式查看 [root@k8s-6-94 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-h5gw5 1/1 Running 0 46s 172.6.95.2 k8s-6-95.host.com <none> <none> nginx-ds-mxjcd 1/1 Running 0 46s 172.6.94.2 k8s-6-94.host.com <none> <none> [root@k8s-6-94 ~]# curl 172.6.94.2 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> [root@k8s-6-94 ~]# curl 172.6.95.2 //curl 不到？ 原因是跨宿主机，docker容器还不能通信，下篇：flannel解决这个问题
3：在运算节点(192.168.6.95)上检查
[root@k8s-6-95 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-ds-h5gw5 1/1 Running 0 109s nginx-ds-mxjcd 1/1 Running 0 109s # 扩展式查看 [root@k8s-6-95 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-h5gw5 1/1 Running 0 2m3s 172.6.95.2 k8s-6-95.host.com <none> <none> nginx-ds-mxjcd 1/1 Running 0 2m3s 172.6.94.2 k8s-6-94.host.com <none> <none> [root@k8s-6-95 ~]# curl 172.6.95.2 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> [root@k8s-6-95 ~]# curl 172.6.94.2 //curl 不到，上述已有解释
4：查看kubernetes集群是否搭建成功
[root@k8s-6-94 ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} [root@k8s-6-94 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-6-94.host.com Ready master,node 3h35m v1.15.4 k8s-6-95.host.com Ready <none> 3h27m v1.15.4 [root@k8s-6-94 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-ds-h5gw5 1/1 Running 0 2m53s nginx-ds-mxjcd 1/1 Running 0 2m53s [root@k8s-6-95 ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} [root@k8s-6-95 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-6-94.host.com Ready master,node 3h51m v1.15.4 k8s-6-95.host.com Ready <none> 3h43m v1.15.4 [root@k8s-6-95 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-ds-h5gw5 1/1 Running 0 18m nginx-ds-mxjcd 1/1 Running 0 18m // 在任意一台node节点上查看都可以
6：FAQ:
1.：kube-proxy启动之后，用不了ipvs模式，查看日志发现如下报错
E1111 12:27:38.031263 23086 server_others.go:259] can't determine whether to use ipvs proxy, error: error getting ipset version, error: executable file not found in $PATH W1111 12:27:38.049355 23086 node.go:113] Failed to retrieve node info: Get https://10.4.7.10:7443/api/v1/nodes/hdss7-21.host.com: EOF yum install ipset -y，用supervisor重启服务，问题解决
接下来，讲解Docker容器跨宿主机之间相互通信
浙公网安备 33010602011771号