【Kubernetes系列一】kubernetes1.12安装
准备工作
1.安装环境信息
10.128.27.128 Master
10.128.27.114 node01
10.128.27.115 node02
10.128.27.116 node03
2.添加解析
~]# cat /etc/hosts
10.128.27.128 master.ilinux.io master
10.128.27.114 nodeO1.ilinux.io node01
10.128.27.115 nodeO2.ilinux.io node02
10.128.27.116 nodeO3.ilinux.io node03
3.时间同步
~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.128.24.6 5.103.139.163 2 u 201 1024 377 0.458 1.440 1.179
4.关闭防火墙和selinux
~]# systemctl stop firewalld.service iptables.service
~]# systemctl disable firewalld.service
~]# systemctl disable iptables.service
~]# setenforce 0
~]# sed -i 's@^\(SELINUX=\).*@\1disabled@' /etc/sysconfig/selinux
5.禁用swap设备
~]# swapoff -a
~]# vi /etc/fstab #注释swap行
6.启用ipvs模块(所有节点都需要)
kubernetes1.11后支持ipvs代理模式的service资源，但依赖于ipvs的相关内核模块
~]# 执行脚本
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_mods_dir | grep -o "^[^.]*"); do
/sbin/modinfo -F filename $i &>/dev/null
if [ $? -eq 0 ]; then
/sbin/modprobe $i
fi
done
~]# chmod +x /etc/sysconfig/modules/ipvs.modules
~]# /etc/sysconfig/modules/ipvs.modules
7.验证网络k8s依赖(所有节点)
#永久生效
~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
EOF
sysctl --system
部署kubernetes集群
1.安装docker
wget https://download.docker.com/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install -y docker-ce-18.06.0*
2.开启iptables的转发
~]# vi /usr/lib/systemd/system/docker.service
ExecStartPost= /usr/sbin/iptables -P FORWARD ACCEPT
~]# systemctl daemon-reload
~]# systemctl enable docker.service
~]# systemctl start docker.service
3.修改docker镜像源
~]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}
4.设置kubernetes的yum源
~]# vi /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl= https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey= https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
~]# yum -y install kubernetes-cni-0.6.0-0.x86_64 kubeadm-1.12.2-0.x86_64 kubelet-1.12.2-0.x86_64 kubectl-1.12.2-0.x86_64
5.修改kubelet的配置文件，禁止swap
~]# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
~]# systemctl enable kubelet.service
6.集群初始化
集群初始化有两种方式，一是通过命令行传递参数，一种是基于yaml格式的专用配置文件设定更详细的配置参数
这里采用第二种方式
~]# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.12.2
api:
advertiseAddress: 10.128.27.128
bindPort: 6443
controlPlaneEndpoint: ""
#imageRepository: k8s.gcr.io
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kubeProxy:
config:
mode: "ipvs"
ipvs:
ExcludeCIDRs: null
minSyncPeriod: 0s
scheduler: ""
syncPeriod: 30s
kubeletConfiguration:
baseConfig:
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
failSwapOn: false
resolvConf: /etc/resolv.conf
staticPodPath: /etc/kubernetes/manifests
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
拉取需要用到的镜像，并重新打标签
kubeadm config images pull --config kubeadm-config.yaml
#如果报错可以手动拉取，命令如下
docker pull mirrorgooglecontainers/kube-apiserver-amd64:v1.12.2
docker pull mirrorgooglecontainers/kube-controller-manager-amd64:v1.12.2
docker pull mirrorgooglecontainers/kube-scheduler-amd64:v1.12.2
docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.12.2
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd-amd64:3.2.24
docker pull coredns/coredns:1.2.2
docker tag mirrorgooglecontainers/kube-apiserver-amd64:v1.12.2 k8s.gcr.io/kube-apiserver-amd64:v1.12.2
docker tag mirrorgooglecontainers/kube-controller-manager-amd64:v1.12.2 k8s.gcr.io/kube-controller-manager-amd64:v1.12.2
docker tag mirrorgooglecontainers/kube-scheduler-amd64:v1.12.2 k8s.gcr.io/kube-scheduler-amd64:v1.12.2
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.12.2 k8s.gcr.io/kube-proxy-amd64:v1.12.2
docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag mirrorgooglecontainers/etcd-amd64:3.2.24 k8s.gcr.io/etcd-amd64:3.2.24
docker tag coredns/coredns:1.2.2 k8s.gcr.io/coredns:1.2.2
进行master初始化
[root@master ~]# kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=Swap
[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[preflight] WARNING: unable to stop the kubelet service momentarily: [exit status 5]
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [master.ilinux.io localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [master.ilinux.io localhost] and IPs [10.128.27.128 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [master.ilinux.io kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.128.27.128]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 30.503451 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node master.ilinux.io as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node master.ilinux.io as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master.ilinux.io" as an annotation
[bootstraptoken] using token: vaarra.fkz34e2x998oun00
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 10.128.27.128:6443 --token vaarra.fkz34e2x998oun00 --discovery-token-ca-cert-hash sha256:ef71440837260e80572278e417b478c239c27304e1fd43f536b17ffd34fc4331
7.设定集群配置文件
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
8.查看集群各组件当前状态
~]# kubectl get cs
查看集群节点信息
~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master.ilinux.io NotReady master 66m v1.12.2
因为集群未安装网络插件，所以显示NotReady
9.安装网络插件
为kubernetes提供pod网络的插件有很多，目前最流行的是flannel和calico，这里以flannel为主
~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml #此yaml文件为官方一直更新，由于安装老版本需要官网查询使用k8s版本的yaml文件
ps：
2021年9月8号再次安装集群，到此步骤出现问题，安装旧版本集群使用新版flannel yaml文件后pod起不来。
解决办法：
1、查看官网显示 Kubernetes v1.6~v1.15 可以适用以下yaml文件
官网：https://github.com/flannel-io/flannel/blob/master/Documentation/kubernetes.md
2、更换yaml文件：
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-legacy.yml
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
3、依然pod起不来，显示节点不匹配，但是nodeSelector配置没问题。后续排查发现是因为节点网络插件未安装所以存在not-ready污点，flannel无法调度。
~]# kubectl describe node master.ilinux.io #其他nodes节点存在同样的污点
Taints: node.kubernetes.io/not-ready:NoSchedule
4、修改flannel的yaml文件，重新发布，让flannel yaml可以容忍污点配置。
~]# vi kube-flannel-legacy.yml
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoSchedule
5、发布成功
查看网络插件flanner运行情况
~]# kubectl get pods -n kube-system|grep flannel
再次查看节点信息，发现已经Ready
~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master.ilinux.io Ready master 66m v1.12.2
10.添加node到集群中
# 使用的是集群安装成功后给到的token信息
~]# kubeadm join 10.128.27.128:6443 --ignore-preflight-errors=Swap --token vaarra.fkz34e2x998oun00 --discovery-token-ca-cert-hash sha256:ef71440837260e80572278e417b478c239c27304e1fd43f536b17ffd34fc4331
~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master.ilinux.io Ready master 6h54m v1.12.2
node01.ilinux.io NotReady <none> 5h15m v1.12.2
如果显示NotReady
~]# kubectl describe pod kube-flannel-ds-amd64-n62wg -n kube-system #查看报错信息
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m37s default-scheduler Successfully assigned kube-system/kube-flannel-ds-amd64-n62wg to node01.ilinux.io
Warning FailedCreatePodSandBox 72s (x3 over 7m39s) kubelet, node01.ilinux.io Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 108.177.97.82:443: connect: connection refused
Warning FailedCreatePodSandBox 19s (x18 over 9m21s) kubelet, node01.ilinux.io Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
node拉取不到镜像，不能访问k8s.gcr.io，手动拉取镜像
~]# docker pull mirrorgooglecontainers/pause:3.1
~]# docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master.ilinux.io Ready master 7h42m v1.12.2
node01.ilinux.io Ready <none> 6h4m v1.12.2
node02.ilinux.io Ready <none> 38m v1.12.2
node03.ilinux.io Ready <none> 6s v1.12.2
11.获取集群状态信息
~]# kubectl cluster-info
12.获取集群版本信息
~]# kubectl version --short=true
从集群中移除节点
1.迁移节点上的pod资源到其他节点
~]# kubectl drain NODE_ID --delete-local-data --force --ignore-daemonsets
~]# kubectl delete node NODE_ID
2.在要删除的node上执行如下命令重置系统状态
~]# kubeadm reset
浙公网安备 33010602011771号