kubernetes第五次课
1. 总结 Underlay 和 Overlay 网络的的区别及优缺点
1.1 Overlay 网络
1.1.1 Overlay网络简介
Overlay叫叠加网络也叫覆盖网络，指的是在物理网络的基础之上叠加实现新的虚拟网络，即可使网络的中的容器可以相互通信。
优点是对物理网络的兼容性比较好，可以实现pod的夸宿主机子网通信。
calico与flannel等网络插件都支持overlay网络。
缺点是有额外的封装与解封性能开销。
目前私有云使用比较多。
1.1.2 overlay设备简介
VTEP(VXLAN Tunnel Endpoint vxlan隧道端点),VTEP是VXLAN网络的边缘设备，是VXLAN隧道的起点和终点，VXLAN对用户原始数据帧的封装和解封装均在VTEP上进行,用于VXLAN报文的封装和解封装,VTEP与物理网络相连,VXLAN报文中源IP地址为本节点的VTEP地址，VXLAN报文中目的IP地址为对端节点的VTEP地址，一对VTEP地址就对应着一个VXLAN隧道，服务器上的虚拟交换机(隧道flannel.1就是VTEP)，比如一个虚拟机网络中的多个vxlan就需要多个VTEP对不同网络的报文进行封装与解封装。
VNI（VXLAN Network Identifier）：VXLAN网络标识VNI类似VLAN ID,用于区分VXLAN段,不同VXLAN段的虚拟机不能直接二层相互通信,一个VNI表示一个租户,即使多个终端用户属于同一个VNI,也表示一个租户。
NVGRE：Network Virtualization using Generic Routing Encapsulation，主要支持者是Microsoft，与VXLAN不同的是，NVGRE没有采用标准传输协议（TCP/UDP），而是借助通用路由封装协议（GRE），NVGRE使用GRE头部的第24位作为租户网络标识符（TNI）， 与VXLAN一样可以支持1777216个vlan。
1.1.3 overlay通信简介
VM A发送L2 帧与VM请求与VM B通信。
源宿主机VTEP添加或者封装VXLAN、UDP及IP头部报文。
网络层设备将封装后的报文通过标准的报文在三层网络进行转发到目标主机。
目标宿主机VTEP删除或者解封装VXLAN、UDP及IP头部。
将原始L2帧发送给目标VM。
1.2 Underlay 网络
Underlay网络就是传统IT基础设施网络，由交换机和路由器等设备组成，借助以太网协议、路由协议和VLAN协议等驱动，它还是Overlay网络的底层网络，为Overlay网络提供数据通信服务。容器网络中的Underlay网络是指借助驱动程序将宿主机的底层网络接口直接暴露给容器使用的一种网络构建技术，较为常见的解决方案有MAC VLAN、IP VLAN和直接路由等。
Underlay依赖于物理网络进行跨主机通信。
kubeadm安装k8s 实现underlay网络
root@k8S-master:~# apt update
root@k8S-master:~# apt -y install apt-transport-https ca-certificates curl software-properties-common
安装GPG证书
root@k8S-master:~# curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
root@k8S-master:~# sudo add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
root@k8S-master:~# apt-get -y update
查看可安装的Docker版本
root@k8S-master:~# apt-cache madison docker-ce docker-ce-cli
root@k8S-master:~# apt install -y docker-ce=5:20.10.23~3-0~ubuntu-jammy docker-ce-cli=5:20.10.23~3-0~ubuntu-jammy
镜像加速
root@k8S-master:/etc/docker# cat daemon.json
{
"registry-mirrors": ["https://ydwu9nbq.mirror.aliyuncs.com"]
}
root@k8S-master:/etc/docker# systemctl restart docker
root@k8S-master:/etc/docker# sudo systemctl daemon-reload
root@k8S-master:/etc/docker# systemctl restart docker
安装cri-dockerd-二进制:
root@k8S-master:~# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd-0.3.1.amd64.tgz
root@k8S-master:~# tar xvf cri-dockerd-0.3.1.amd64.tgz
cri-dockerd/
cri-dockerd/cri-dockerd
root@k8S-master:~# cp cri-dockerd/cri-dockerd /usr/local/bin/
配置cri-dockerd.service文件：
安装cri-dockerd-编写cri-docker.service:
配置cri-dockerd.service文件：
安装cri-dockerd-编写cri-docker.socket:
配置cri-docker.socket文件：
root@k8S-master:~# vim /lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
root@k8S-master:~# vim /etc/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
systemctl daemon-reload && systemctl restart cri-docker && systemctl enable cri-docker && systemctl enable --now cri-docker.socket
安装kubeadm
root@k8S-master:~# apt-get update && apt-get install -y apt-transport-https
root@k8S-master:~# curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
root@k8S-master:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
root@k8S-master:~# apt update
查看可kubeadm安装的版本
root@k8S-master:~# apt-cache madison kubeadm
root@k8S-master:~# apt-get install kubelet=1.24.10-00 kubeadm=1.24.10-00 kubectl=1.24.10-00
root@k8S-master:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.10", GitCommit:"5c1d2d4295f9b4eb12bfbf6429fdf989f2ca8a02", GitTreeState:"clean", BuildDate:"2023-01-18T19:13:52Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
root@k8S-master:~# kubeadm config images list --kubernetes-version v1.24.10
registry.k8s.io/kube-apiserver:v1.24.10
registry.k8s.io/kube-controller-manager:v1.24.10
registry.k8s.io/kube-scheduler:v1.24.10
registry.k8s.io/kube-proxy:v1.24.10
registry.k8s.io/pause:3.7
registry.k8s.io/etcd:3.5.6-0
registry.k8s.io/coredns/coredns:v1.8.6
root@k8S-master:~# vim images-download.sh
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.6-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.6
root@k8S-master:~# kubeadm init --apiserver-advertise-address=10.4.7.136 \
--apiserver-bind-port=6443 \
--kubernetes-version=v1.24.10 \
--pod-network-cidr=10.200.0.0/16 \
--service-cidr=10.4.5.0/24 \
--service-dns-domain=cluster.local \
--image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
--ignore-preflight-errors=swap \
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.4.7.136:6443 --token ndy56z.xpqrcqqlndywsg3g \
--discovery-token-ca-cert-hash sha256:7e345047689b53d3516e896b6b75f5b08a3d8cd134e66f7b98bc269bfa74b0ab
root@k8S-master:~# mkdir -p $HOME/.kube
root@k8S-master:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config
root@k8S-master:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config
root@k8S-node01:~# kubeadm join 10.4.7.136:6443 --token ndy56z.xpqrcqqlndywsg3g --discovery-token-ca-cert-hash sha256:7e345047689b53d3516e896b6b75f5b08a3d8cd134e66f7b98bc269bfa74b0ab --cri-socket unix:///var/run/cri-dockerd.sock
root@k8S-master:~# scp /root/.kube/config 10.4.7.137:/root/.kube/
root@k8S-master:~# scp /root/.kube/config 10.4.7.138:/root/.kube/
root@k8S-master:~# scp /root/.kube/config 10.4.7.139:/root/.kube/
root@k8S-master:~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master.awen.com NotReady control-plane 7m42s v1.24.10
k8s-node01.awen.com NotReady <none> 5m29s v1.24.10
k8s-node02.awen.com NotReady <none> 3m4s v1.24.10
k8s-node03.awen.com NotReady <none> 2m38s v1.24.10
root@k8S-node01:~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master.awen.com NotReady control-plane 7m44s v1.24.10
k8s-node01.awen.com NotReady <none> 5m31s v1.24.10
k8s-node02.awen.com NotReady <none> 3m6s v1.24.10
k8s-node03.awen.com NotReady <none> 2m40s v1.24.10
root@k8S-node02:~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master.awen.com NotReady control-plane 7m47s v1.24.10
k8s-node01.awen.com NotReady <none> 5m34s v1.24.10
k8s-node02.awen.com NotReady <none> 3m9s v1.24.10
k8s-node03.awen.com NotReady <none> 2m43s v1.24.10
root@k8s-node03:~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master.awen.com NotReady control-plane 7m50s v1.24.10
k8s-node01.awen.com NotReady <none> 5m37s v1.24.10
k8s-node02.awen.com NotReady <none> 3m12s v1.24.10
k8s-node03.awen.com NotReady <none> 2m46s v1.24.10
安装helm
下载二进制安装包
root@k8S-master:~# wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
root@k8S-master:~# tar xvf helm-v3.9.0-linux-amd64.tar.gz
linux-amd64/
linux-amd64/helm
linux-amd64/LICENSE
linux-amd64/README.md
root@k8S-master:~# mv linux-amd64/helm /usr/local/bin/
部署网络组件hybridnet
root@k8S-master:~# helm repo add hybridnet https://alibaba.github.io/hybridnet/
"hybridnet" has been added to your repositories
root@k8S-master:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hybridnet" chart repository
Update Complete. ⎈Happy Helming!⎈
root@k8S-master:~# helm install hybridnet hybridnet/hybridnet -n kube-system --set init.cidr=10.200.0.0/16(pod地址) pod地址为overlay
root@k8S-master:~# helm install hybridnet hybridnet/hybridnet -n kube-system --set init.cidr=10.200.0.0/16
W0303 21:47:26.933166 22578 warnings.go:70] spec.template.spec.nodeSelector[beta.kubernetes.io/os]: deprecated since v1.14; use "kubernetes.io/os" instead
W0303 21:47:26.933182 22578 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
NAME: hybridnet
LAST DEPLOYED: Fri Mar 3 21:47:25 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
root@k8S-master:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-typha-6f55876f98-grfch 0/1 ContainerCreating 0 42s
kube-system calico-typha-6f55876f98-mvzhm 0/1 ContainerCreating 0 42s
kube-system calico-typha-6f55876f98-vs925 0/1 ContainerCreating 0 42s
kube-system coredns-7f74c56694-8kxqx 0/1 Pending 0 32m
kube-system coredns-7f74c56694-9x5kd 0/1 Pending 0 32m
kube-system etcd-k8s-master.awen.com 1/1 Running 0 32m
kube-system hybridnet-daemon-4gm6v 0/2 Init:0/1 0 42s
kube-system hybridnet-daemon-6lvqh 0/2 Init:0/1 0 43s
kube-system hybridnet-daemon-7z8lr 0/2 lnit:0/1 0 43s
kube-system hybridnet-daemon-jt4rr 0/2 Init:0/1 0 43s
kube-system hybridnet-manager-6574dcc5fb-fwc2d 0/1 Pending 0 42s
kube-system hybridnet-manager-6574dcc5fb-g2b9g 0/1 Pending 0 42s
kube-system hybridnet-manager-6574dcc5fb-tsrm8 0/1 Pending 0 42s
kube-system hybridnet-webhook-76dc57b4bf-r9nlw 0/1 Pending 0 43s
kube-system hybridnet-webhook-76dc57b4bf-zdfw7 0/1 Pending 0 43s
kube-system hybridnet-webhook-76dc57b4bf-zsqfr 0/1 Pending 0 43s
kube-system kube-apiserver-k8s-master.awen.com 1/1 Running 0 32m
kube-system kube-controller-manager-k8s-master.awen.com 1/1 Running 0 32m
kube-system kube-proxy-42bfd 1/1 Running 0 30m
kube-system kube-proxy-mcsw4 1/1 Running 0 27m
kube-system kube-proxy-tkd6r 1/1 Running 0 28m
kube-system kube-proxy-z9j66 1/1 Running 0 32m
kube-system kube-scheduler-k8s-master.awen.com 1/1 Running 0 32m
root@k8S-master:~# kubectl describe pod hybridnet-manager-6574dcc5fb-fwc2d -n kube-system
Warning FailedScheduling 80s default-scheduler 0/4 nodes are available: 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
root@k8S-master:~# kubectl label node k8s-node02.awen.com node-role.kubernetes.io/master=
root@k8S-master:~# kubectl label node k8s-node03.awen.com node-role.kubernetes.io/master=
root@k8S-master:~# kubectl label node k8s-node01.awen.com node-role.kubernetes.io/master=
root@k8S-master:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-typha-6f55876f98-5m7lb 1/1 Running 0 57m
kube-system calico-typha-6f55876f98-7m9fl 1/1 Running 0 57m
kube-system calico-typha-6f55876f98-rpd2n 1/1 Running 0 57m
kube-system coredns-7f74c56694-9x5kd 1/1 Running 0 112m
kube-system coredns-7f74c56694-v8znb 1/1 Running 0 59m
kube-system etcd-k8s-master.awen.com 1/1 Running 0 112m
kube-system hybridnet-daemon-b7cs8 2/2 Running 1 (55m ago) 57m
kube-system hybridnet-daemon-nt2r7 2/2 Running 1 (54m ago) 57m
kube-system hybridnet-daemon-wt7lh 2/2 Running 0 65s
kube-system hybridnet-daemon-zrk88 2/2 Running 0 57m
kube-system hybridnet-manager-6574dcc5fb-57dvg 1/1 Running 0 57m
kube-system hybridnet-manager-6574dcc5fb-8bmhd 1/1 Running 2 (60s ago) 2m56s
kube-system hybridnet-manager-6574dcc5fb-dmrpp 1/1 Running 0 57m
kube-system hybridnet-webhook-76dc57b4bf-45dzh 1/1 Running 2 (60s ago) 5m13s
kube-system hybridnet-webhook-76dc57b4bf-b6zr9 1/1 Running 0 57m
kube-system hybridnet-webhook-76dc57b4bf-ds8vf 1/1 Running 0 57m
kube-system kube-apiserver-k8s-master.awen.com 1/1 Running 0 112m
kube-system kube-controller-manager-k8s-master.awen.com 1/1 Running 0 112m
kube-system kube-proxy-42bfd 1/1 Running 0 110m
kube-system kube-proxy-mcsw4 1/1 Running 1 (2m16s ago) 107m
kube-system kube-proxy-tkd6r 1/1 Running 0 107m
kube-system kube-proxy-z9j66 1/1 Running 0 112m
kube-system kube-scheduler-k8s-master.awen.com 1/1 Running 0 112m
root@k8S-master:~# cat 1.create-underlay-network.yaml
---
apiVersion: networking.alibaba.com/v1
kind: Network
metadata:
name: underlay-network1
spec:
netID: 0
type: Underlay
nodeSelector:
network: "underlay-nethost"
---
apiVersion: networking.alibaba.com/v1
kind: Subnet
metadata:
name: underlay-network1
spec:
network: underlay-network1
netID: 0
range:
version: "4"
cidr: "10.4.7.0/24"
gateway: "10.4.7.254" # 外部网关地址
start: "10.4.7.1"
end: "10.4.7.254"
root@k8S-master:~# kubectl apply -f 1.create-underlay-network.yaml
network.networking.alibaba.com/underlay-network1 created
subnet.networking.alibaba.com/underlay-network1 created
验证网络
root@k8S-master:~# kubectl get network
NAME NETID TYPE MODE V4TOTAL V4USED V4AVAILABLE LASTALLOCATEDV4SUBNET V6TOTAL V6USED V6AVAILABLE LASTALLOCATEDV6SUBNET
init 4 Overlay 65534 2 65532 init 0 0 0
underlay-network1 0 Underlay 253 0 253 underlay-network1 0 0 0
创建pod并使用overlay网络 (不做修改)
目前 默认会使用overlay网络，overlay使用的是vxlan进行报文封装，会生成一个eth0.vxlan4网卡，接收报文为UDP的8472端口
root@k8S-master:~# kubectl create ns myserver
namespace/myserver created
root@k8S-master:~# kubectl apply -f 2.tomcat-app1-overlay.yaml
deployment.apps/myserver-tomcat-app1-deployment-overlay created
service/myserver-tomcat-app1-service-overlay created
root@k8S-master:~# kubectl get pod,svc -o wide -n myserver -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z 1/1 Running 0 43s 10.200.0.6 k8s-node02.awen.com <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/myserver-tomcat-app1-service-overlay NodePort 10.4.5.68 <none> 80:30003/TCP 43s app=myserver-tomcat-app1-overlay-selector
root@k8S-master:~# kubectl exec -it myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z bash -n myserver
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z /]# ping www.baidu.com
PING www.a.shifen.com (110.242.68.3) 56(84) bytes of data.
64 bytes from 110.242.68.3 (110.242.68.3): icmp_seq=1 ttl=127 time=22.2 ms
64 bytes from 110.242.68.3 (110.242.68.3): icmp_seq=2 ttl=127 time=28.1 ms
64 bytes from 110.242.68.3 (110.242.68.3): icmp_seq=3 ttl=127 time=22.0 ms
^C
--- www.a.shifen.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 22.063/24.135/28.101/2.807 ms
[root@myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z /]#
root@k8S-master:~# kubectl exec -it myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z bash -n myserver
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z /]# ping www.baidu.com
PING www.a.shifen.com (110.242.68.3) 56(84) bytes of data.
64 bytes from 110.242.68.3 (110.242.68.3): icmp_seq=1 ttl=127 time=22.2 ms
64 bytes from 110.242.68.3 (110.242.68.3): icmp_seq=2 ttl=127 time=28.1 ms
64 bytes from 110.242.68.3 (110.242.68.3): icmp_seq=3 ttl=127 time=22.0 ms
^C
--- www.a.shifen.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 22.063/24.135/28.101/2.807 ms
[root@myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z /]#
创建pod并使用underlay网络
root@k8S-master:~# vi 3.tomcat-app1-underlay.yaml
18 annotations: #使用Underlay或者Overlay网络
19 networking.alibaba.com/network-type: Underlay 定义网络
root@k8S-master:~# kubectl apply -f 3.tomcat-app1-underlay.yaml
deployment.apps/myserver-tomcat-app1-deployment-underlay created
service/myserver-tomcat-app1-service-underlay created
root@k8S-master:~# kubectl get pod -n myserver -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myserver-tomcat-app1-deployment-overlay-7864dd6bdb-pzh6z 1/1 Running 0 29m 10.200.0.6 k8s-node02.awen.com <none> <none>
myserver-tomcat-app1-deployment-underlay-5f7dd46d56-ktq2k 1/1 Running 0 10s 10.4.7.2 k8s-node02.awen.com <none> <none>
有状态应用可以使用underlay网络
通过service IP访问Pod
root@k8S-master:~# kubectl get svc -n myserver
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
myserver-tomcat-app1-service-overlay NodePort 10.4.5.68 <none> 80:30003/TCP 48m
myserver-tomcat-app1-service-underlay ClusterIP 10.4.5.52 <none> 80/TCP 19m
windown系统
管理员运行cmd
C:\WINDOWS\system32>route add 10.4.5.0 MASK 255.255.255.0 -p 10.4.7.138
使用service请求是因为 使用service可以使用轮询的方式把请求转发给后面的pod
Flannel、calico通信模式
Flannel vxlan架构图：
源pod发起请求，此时报文中源IP为pod的eth0 的ip,源mac 为pod的eth0的mac，目的Pod为目的Pod的IP，目的mac为网关(cni0)的MAC，
数据报文通过veth peer发送给网关cni0,检查目的mac就是发给自己的，cni0进行目标IP检查,如果是同一个网桥的报文就直接转发，不是的话就发送给
flannel.1，宿主机基于UDP封装Vxlan报文，报文到达目的宿主机物理网卡，接开始解封装报文，报文到达目的宿主机flannel.1，flannel.1检查报文的目的IP，发现是去往本机cni0的子网，将请求报文转发至cni0，cni0基于目的IP检查mac地址表，修改目的MAC为目的MAC后将来请求转给pod报文到达目的宿主机cni0
发送：pod eth0->host veth->host cni0->host vxlan(flannel.1)->host eth0
接收：host eth0->host flannel.1-> host cni0 -> host veth0 -> pod eth0
calico
源pod发起请求, 报文到达宿主机与pod对应的网卡，，报文到达源宿主机tun0，报文到达目的宿主机eth0，报文到达目的宿主机eth0，报文到达目的宿主机tunl0，报文到达目的Pod与目的宿主机对应的网卡，报文到达目的Pod
发送：pod eth0->host cali -> host tun0 -> host eth0
接收：host eth0->host tun0 ->host cali -> pod eth0
扩展
1. 基于二进制实现高可用的 K8S 集群环境
安装harbor
安装docker docker-compose
root@awen:~# hostnamectl set-hostname k8s-harbor.awen.com
root@awen:~# bash
root@k8s-harbor:~#
root@k8s-harbor:~# wget https://github.com/goharbor/harbor/releases/download/v2.6.3/harbor-offline-installer-v2.6.3.tgz
root@k8s-harbor:~# mkdir /apps
root@k8s-harbor:~# mv /root/harbor-offline-installer-v2.6.3.tgz /apps
root@k8s-harbor:/apps# tar zxvf harbor-offline-installer-v2.6.3.tgz
harbor/harbor.v2.6.3.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
root@k8s-harbor:/apps# cd harbor
root@k8s-harbor:/apps/harbor# cp harbor.yml.tmpl harbor.yml
root@k8s-harbor:/apps/harbor# vi harbor.yml
hostname: www.awen.com
root@k8s-harbor:/apps/harbor# mkdir certs
root@k8s-harbor:/apps/harbor# cd certs/
⾃签发证书
root@k8s-harbor:/apps/harbor/certs# openssl genrsa -out harbor-ca.key
root@k8s-harbor:/apps/harbor/certs# openssl req -x509 -new -nodes -key harbor-ca.key -subj "/CN=www.awen.com" -days 7120 -out harbor-ca.crt
root@k8s-harbor:/apps/harbor# vi harbor.yml
17 certificate: /apps/harbor/certs/harbor-ca.crt
18 private_key: /apps/harbor/certs/harbor-ca.key
挂载一款数据盘
53 data_volume: /data
root@k8s-harbor:~# fdisk -l
Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors
root@k8s-harbor:~# mkfs.xfs /dev/sdb
meta-data=/dev/sdb isize=512 agcount=4, agsize=655360 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@k8s-harbor:~# mkdir /data/harbor -p
root@k8s-harbor:~# vi /etc/fstab
/dev/sdb /data/harbor xfs defaults 0 0
root@k8s-harbor:~# mount -a
root@k8s-harbor:~# df -TH
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 408M 1.6M 407M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv ext4 26G 9.7G 15G 41% /
tmpfs tmpfs 2.1G 0 2.1G 0% /dev/shm
tmpfs tmpfs 5.3M 0 5.3M 0% /run/lock
/dev/sda2 ext4 2.1G 135M 1.8G 8% /boot
tmpfs tmpfs 408M 4.1k 408M 1% /run/user/0
/dev/sdb xfs 11G 109M 11G 2% /data/harbor
root@k8s-harbor:/apps/harbor# vi harbor.yml
data_volume: /data/harbor
安装harbor
root@k8s-harbor:/apps/harbor# ./install.sh --with-trivy --with-chartmuseum
/etc/hosts文件
10.4.7.131 www.awen.com
创建证书
生成私钥
openssl genrsa -out awen.com.key 4096
生成证书签名请求
openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=awen.com" \
-key awen.com.key \
-out awen.com.csr
生成 x509 v3 扩展文件。
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=awen.com
DNS.2=awen
DNS.3=k8s-harbor.awen.com
EOF
openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in awen.com.csr \
-out awen.com.crt
登录harbor
登录harbor成功
root@k8s-harbor:/apps/harbor# docker login www.awen.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
跨主机上传镜像
root@k8s-ha:~# mkdir -p /etc/docker/certs.d/www.awen.com
root@k8s-harbor:/apps/crest# scp awen.com.crt 10.4.7.132:/etc/docker/certs.d/www.awen.com
root@k8s-ha:~# vi /etc/hosts
10.4.7.131 www.awen.com
root@k8s-ha:~# docker login www.awen.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
扩展
1. 基于二进制实现高可用的 K8S 集群环境
安装harbor
安装docker docker-compose
root@awen:~# hostnamectl set-hostname k8s-harbor.awen.com
root@awen:~# bash
root@k8s-harbor:~#
root@k8s-harbor:~# wget https://github.com/goharbor/harbor/releases/download/v2.6.3/harbor-offline-installer-v2.6.3.tgz
root@k8s-harbor:~# mkdir /apps
root@k8s-harbor:~# mv /root/harbor-offline-installer-v2.6.3.tgz /apps
root@k8s-harbor:/apps# tar zxvf harbor-offline-installer-v2.6.3.tgz
harbor/harbor.v2.6.3.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
root@k8s-harbor:/apps# cd harbor
root@k8s-harbor:/apps/harbor# cp harbor.yml.tmpl harbor.yml
root@k8s-harbor:/apps/harbor# vi harbor.yml
5 hostname: www.awen.com
6
7 # http related config
8 http:
9 # port for http, default is 80. If https enabled, this port will redirect to https port
10 port: 80
11
12 # https related config
13 https:
14 # https port for harbor, default is 443
15 port: 443
16 # The path of cert and key files for nginx
17 certificate: /apps/crest/awen.com.crt
18 private_key: /apps/crest/awen.com.key
挂载一款数据盘
47 data_volume: /data
root@k8s-harbor:~# fdisk -l
Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors
root@k8s-harbor:~# mkfs.xfs /dev/sdb
meta-data=/dev/sdb isize=512 agcount=4, agsize=655360 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@k8s-harbor:~# mkdir /data/harbor -p
root@k8s-harbor:~# vi /etc/fstab
/dev/sdb /data/harbor xfs defaults 0 0
root@k8s-harbor:~# mount -a
root@k8s-harbor:~# df -TH
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 408M 1.6M 407M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv ext4 26G 9.7G 15G 41% /
tmpfs tmpfs 2.1G 0 2.1G 0% /dev/shm
tmpfs tmpfs 5.3M 0 5.3M 0% /run/lock
/dev/sda2 ext4 2.1G 135M 1.8G 8% /boot
tmpfs tmpfs 408M 4.1k 408M 1% /run/user/0
/dev/sdb xfs 11G 109M 11G 2% /data/harbor
root@k8s-harbor:/apps/harbor# vi harbor.yml
data_volume: /data/harbor
安装harbor
root@k8s-harbor:/apps/harbor# ./install.sh --with-trivy --with-chartmuseum
/etc/hosts文件
10.4.7.131 www.awen.com
创建证书
生成私钥
openssl genrsa -out ca.key 4096openssl genrsa -out awen.com.key 4096
生成证书签名请求
openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=awen.com" \
-key awen.com.key \
-out awen.com.csr
生成 x509 v3 扩展文件。
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=awen.com
DNS.2=awen
DNS.3=k8s-harbor.awen.com
EOF
openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in awen.com.csr \
-out awen.com.crt
完整创建证书
生成 CA 证书私钥
openssl genrsa -out ca.key 4096
生成CA证书
openssl req -x509 -new -nodes -sha512 -days 3650 \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=awen.com" \
-key ca.key \
-out ca.crt
生成私钥
openssl genrsa -out awen.com.key 4096
生成证书签名请求
openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=awen.com" \
-key awen.com.key \
-out awen.com.csr
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=www.awen.com
DNS.2=awen
DNS.3=k8s-harbor.awen.com
EOF
openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in awen.com.csr \
-out awen.com.crt
安装ha服务
root@k8s-ha:~# apt install keepalived haproxy
root@k8s-ha:~# cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/keepalived.conf
root@k8s-ha:/etc/keepalived# vi keepalived.conf
root@k8s-ha:/etc/keepalived# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
garp_master_delay 10
smtp_alert
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.4.7.50 label eth0:0
10.4.7.51 label eth0:1
10.4.7.52 label eth0:2
10.4.7.53 label eth0:3
10.4.7.54 label eth0:4
}
}
root@k8s-ha:/etc/keepalived# systemctl restart keepalived.service
root@k8s-ha:/etc/keepalived# systemctl enable keepalived.service
root@k8s-ha:/etc/keepalived# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.7.132 netmask 255.255.255.0 broadcast 10.4.7.255
inet6 fe80::20c:29ff:fea8:e812 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:a8:e8:12 txqueuelen 1000 (Ethernet)
RX packets 9083 bytes 7288491 (7.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5445 bytes 635716 (635.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.7.50 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:a8:e8:12 txqueuelen 1000 (Ethernet)
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.7.51 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:a8:e8:12 txqueuelen 1000 (Ethernet)
eth0:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.7.52 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:a8:e8:12 txqueuelen 1000 (Ethernet)
eth0:3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.7.53 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:a8:e8:12 txqueuelen 1000 (Ethernet)
eth0:4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.7.54 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:a8:e8:12 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 129 bytes 11508 (11.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 129 bytes 11508 (11.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@k8s-ha:/etc/keepalived# vi /etc/haproxy/haproxy.cfg
listen k8s_api_nodes_6443
bind 10.4.7.50:6443
mode tcp
#balance leastconn
server 10.4.7.50 10.4.7.50:6443 check inter 2000 fall 3 rise 5
server 10.4.7.51 10.4.7.51:6443 check inter 2000 fall 3 rise 5
server 10.4.7.52 10.4.7.52:6443 check inter 2000 fall 3 rise 5
root@k8s-ha:/etc/keepalived# ss -ltn
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 10.4.7.50:6443 0.0.0.0:*
跨主机上传镜像
root@k8s-ha:~# mkdir -p /etc/docker/certs.d/www.awen.com
root@k8s-harbor:/apps/crest# scp awen.com.crt 10.4.7.132:/etc/docker/certs.d/www.awen.com
root@k8s-ha:~# vi /etc/hosts
10.4.7.131 www.awen.com
root@k8s-ha:~# docker login https://www.awen.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
上传镜像
root@k8s-ha:~# docker pull centos:7.9.2009
7.9.2009: Pulling from library/centos
2d473b07cdd5: Pull complete
Digest: sha256:9d4bcbbb213dfd745b58be38b13b996ebb5ac315fe75711bd618426a630e0987
Status: Downloaded newer image for centos:7.9.2009
docker.io/library/centos:7.9.2009
root@k8s-ha:~# docker tag docker.io/library/centos:7.9.2009 www.awen.com/baseimages/centos:7.9.2009
root@k8s-ha:~# docker push www.awen.com/baseimages/centos:7.9.2009
The push refers to repository [www.awen.com/baseimages/centos]
174f56854903: Pushed
7.9.2009: digest: sha256:dead07b4d8ed7e29e98de0f4504d87e8880d4347859d839686a31da35a3b532f size: 529
安装ansible
root@k8s-ansible:~# apt install python3-pip git -y
root@k8s-ansible:~# pip3 install ansible -i https://mirrors.aliyun.com/pypi/simple/
查看可ansible版本
root@k8s-ansible:~# ansible --version
ansible [core 2.14.3]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (/usr/bin/python3)
jinja version = 3.0.3
libyaml = True
安装sshpass命令⽤于同步公钥到各k8s服务器
root@k8s-ansible:~# apt-get install sshpass -y
10.4.7.131 harbor
10.4.7.132 ha
10.4.7.133 ansible
10.4.7.134 master1
10.4.7.135 master2
10.4.7.136 master3
10.4.7.137 node1
10.4.7.138 node2
10.4.7.139 node3
ansible创建密钥
root@k8s-ansible:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Sa/931jGyog61m0kj2c1XgDr+MuerMi/Xbw48P1kFOg root@k8s-ansible.awen.com
The key's randomart image is:
+---[RSA 3072]----+
| |
| . . |
| . + . |
| . o o . .|
| S .o E ..|
| o+ o.oo.|
| ...X +oo*|
| .o.o*X*oX |
| .++=B@+*.o|
+----[SHA256]-----+
root@k8s-ansible:~#
安装sshpass命令⽤于同步公钥到各k8s服务器
root@k8s-ansible:~# vi scp-key.sh
#!/bin/bash
#目标主机地址
IP="
10.4.7.134
10.4.7.135
10.4.7.136
10.4.7.137
10.4.7.138
10.4.7.139
"
for node in ${IP};do
sshpass -p 000000 ssh-copy-id -i /root/.ssh/id_rsa.pub -o StrictHostKeyChecking=no root@${node}
if [ $? -eq 0 ];then
echo "${node} 秘钥copy完成"
else
echo "${node} 秘钥copy失败"
fi
done
部署节点点下载部署项⽬及组件：
root@k8s-ansible:~# export release=3.2.0
root@k8s-ansible:~# wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown
root@k8s-ansible:~# chmod a+x ezdown
root@k8s-ansible:~# vi ezdown
# default settings, can be overridden by cmd line options, see usage
DOCKER_VER=20.10.19 会检查当前主机的docker版本 当前主机有docker使用当前主机的docker 没有就下载配置文件指定版本
KUBEASZ_VER=3.2.0
K8S_BIN_VER=v1.23.1
EXT_BIN_VER=1.0.0
SYS_PKG_VER=0.4.2
HARBOR_VER=v2.1.3
REGISTRY_MIRROR=CN
root@k8s-ansible:~# ./ezdown
root@k8s-ansible:~# ./ezdown
Usage: ezdown [options] [args]
option: -{DdekSz}
-C stop&clean all local containers
-D download all into "/etc/kubeasz"
-P download system packages for offline installing
-R download Registry(harbor) offline installer
-S start kubeasz in a container
-d <ver> set docker-ce version, default "20.10.19"
-e <ver> set kubeasz-ext-bin version, default "1.0.0"
-k <ver> set kubeasz-k8s-bin version, default "v1.23.1"
-m <str> set docker registry mirrors, default "CN"(used in Mainland,China)
-p <ver> set kubeasz-sys-pkg version, default "0.4.2"
-z <ver> set kubeasz version, default "3.2.0"
先下载镜像，镜像里面存放二进制，使用容器拉起一个容器，把容器的二进制文件拷到/etc/kubeasz 的某个目录 最后回收容器
root@k8s-ansible:~# ./ezdown -D
2023-03-06 04:46:09 INFO Action successed: download_all
root@k8s-ansible:/etc/kubeasz# ./ezctl new k8s-01
2023-03-06 08:02:11 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-01
2023-03-06 08:02:11 DEBUG set versions
2023-03-06 08:02:11 DEBUG cluster k8s-01: files successfully created.
2023-03-06 08:02:11 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-01/hosts'
2023-03-06 08:02:11 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-01/config.yml'
root@k8s-ansible:/etc/kubeasz# cd clusters/
root@k8s-ansible:/etc/kubeasz/clusters# ls
k8s-01
root@k8s-ansible:/etc/kubeasz/clusters# cd k8s-01/
root@k8s-ansible:/etc/kubeasz/clusters/k8s-01# ls
config.yml hosts
root@k8s-ansible:/etc/kubeasz/clusters/k8s-01# vi hosts
批量修改%s/192.168.1/10.4.7/g
root@k8s-ansible:/etc/kubeasz/clusters/k8s-01# cat hosts
# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
10.4.7.134
10.4.7.135
10.4.7.136
# master node(s)
[kube_master]
10.4.7.134
10.4.7.135
# work node(s)
[kube_node]
10.4.7.137
10.4.7.138
# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#10.4.7.8 NEW_INSTALL=false
# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#10.4.7.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#10.4.7.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
# [optional] ntp server for the cluster
[chrony]
#10.4.7.1
[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"
# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"
# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"
# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"
# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.100.0.0/16"
# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.200.0.0/16"
# NodePort Range
NODE_PORT_RANGE="30000-60000"
# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="awen.local"
# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/usr/local/bin"
# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"
# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-01"
# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"
root@k8s-ansible:/etc/kubeasz/clusters/k8s-01# cat config.yml
############################
# prepare
############################
# 可选离线安装系统软件包 (offline|online)
INSTALL_SOURCE: "online"
# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening
OS_HARDEN: false
# 设置时间源服务器【重要：集群内机器时间必须同步】
ntp_servers:
- "ntp1.aliyun.com"
- "time1.cloud.tencent.com"
- "0.cn.pool.ntp.org"
# 设置允许内部时间同步的网络段，比如"10.0.0.0/8"，默认全部允许
local_network: "0.0.0.0/0"
############################
# role:deploy
############################
# default: ca will expire in 100 years
# default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"
# kubeconfig 配置参数
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"
# k8s version
K8S_VER: "1.23.1"
############################
# role:etcd
############################
# 设置不同的wal目录，可以避免磁盘io竞争，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""
############################
# role:runtime [containerd,docker]
############################
# ------------------------------------------- containerd
# [.]启用容器仓库镜像
ENABLE_MIRROR_REGISTRY: true
# [containerd]基础容器镜像
SANDBOX_IMAGE: "easzlab/pause:3.6"
# [containerd]容器持久化存储目录
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"
# ------------------------------------------- docker
# [docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"
# [docker]开启Restful API
ENABLE_REMOTE_API: false
# [docker]信任的HTTP仓库
INSECURE_REG: '["127.0.0.1/8" "www.awen.com" "10.4.7.131"]'
############################
# role:kube-master
############################
# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）
MASTER_CERT_HOSTS:
- "10.4.7.50"
- "api.awen.com"
#- "www.test.com"
# node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）
# 如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24
############################
# role:kube-node
############################
# Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"
# node节点最大pod 数
MAX_PODS: 200
# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量
# 数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"
# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；
# 并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2
# 系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留
# 另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"
# haproxy balance mode
BALANCE_ALG: "roundrobin"
############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
# ------------------------------------------- flannel
# [flannel]设置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false
# [flannel] flanneld_image: "quay.io/coreos/flannel:v0.10.0-amd64"
flannelVer: "v0.15.1"
flanneld_image: "easzlab/flannel:{{ flannelVer }}"
# [flannel]离线镜像tar包
flannel_offline: "flannel_{{ flannelVer }}.tar"
# ------------------------------------------- calico
# [calico]设置 CALICO_IPV4POOL_IPIP=“off”,可以提高网络性能，条件限制详见 docs/setup/calico.md
CALICO_IPV4POOL_IPIP: "Always"
# [calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"
# [calico]设置calico 网络 backend: brid, vxlan, none
CALICO_NETWORKING_BACKEND: "brid"
# [calico]更新支持calico 版本: [v3.3.x] [v3.4.x] [v3.8.x] [v3.15.x]
calico_ver: "v3.19.3"
# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"
# [calico]离线镜像tar包
calico_offline: "calico_{{ calico_ver }}.tar"
# ------------------------------------------- cilium
# [cilium]CILIUM_ETCD_OPERATOR 创建的 etcd 集群节点数 1,3,5,7...
ETCD_CLUSTER_SIZE: 1
# [cilium]镜像版本
cilium_ver: "v1.4.1"
# [cilium]离线镜像tar包
cilium_offline: "cilium_{{ cilium_ver }}.tar"
# ------------------------------------------- kube-ovn
# [kube-ovn]选择 OVN DB and OVN Control Plane 节点，默认为第一个master节点
OVN_DB_NODE: "{{ groups['kube_master'][0] }}"
# [kube-ovn]离线镜像tar包
kube_ovn_ver: "v1.5.3"
kube_ovn_offline: "kube_ovn_{{ kube_ovn_ver }}.tar"
# ------------------------------------------- kube-router
# [kube-router]公有云上存在限制，一般需要始终开启 ipinip；自有环境可以设置为 "subnet"
OVERLAY_TYPE: "full"
# [kube-router]NetworkPolicy 支持开关
FIREWALL_ENABLE: "true"
# [kube-router]kube-router 镜像版本
kube_router_ver: "v0.3.1"
busybox_ver: "1.28.4"
# [kube-router]kube-router 离线镜像tar包
kuberouter_offline: "kube-router_{{ kube_router_ver }}.tar"
busybox_offline: "busybox_{{ busybox_ver }}.tar"
############################
# role:cluster-addon
############################
# coredns 自动安装
dns_install: "no"
corednsVer: "1.8.6"
ENABLE_LOCAL_DNS_CACHE: false
dnsNodeCacheVer: "1.21.1"
# 设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"
# metric server 自动安装
metricsserver_install: "no"
metricsVer: "v0.5.2"
# dashboard 自动安装
dashboard_install: "no"
dashboardVer: "v2.4.0"
dashboardMetricsScraperVer: "v1.0.7"
# ingress 自动安装
ingress_install: "no"
ingress_backend: "traefik"
traefik_chart_ver: "10.3.0"
# prometheus 自动安装
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "12.10.6"
# nfs-provisioner 自动安装
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.1.10"
nfs_path: "/data/nfs"
############################
# role:harbor
############################
# harbor version，完整版本号
HARBOR_VER: "v2.1.3"
HARBOR_DOMAIN: "harbor.yourdomain.com"
HARBOR_TLS_PORT: 8443
# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: false
# install extra component
HARBOR_WITH_NOTARY: false
HARBOR_WITH_TRIVY: false
HARBOR_WITH_CLAIR: false
HARBOR_WITH_CHARTMUSEUM: true
root@k8s-ansible:/etc/kubeasz# ./ezctl --help
Usage: ezctl COMMAND [args]
-------------------------------------------------------------------------------------
Cluster setups:
list to list all of the managed clusters
checkout <cluster> to switch default kubeconfig of the cluster
new <cluster> to start a new k8s deploy with name 'cluster'
setup <cluster> <step> to setup a cluster, also supporting a step-by-step way
start <cluster> to start all of the k8s services stopped by 'ezctl stop'
stop <cluster> to stop all of the k8s services temporarily
upgrade <cluster> to upgrade the k8s cluster
destroy <cluster> to destroy the k8s cluster
backup <cluster> to backup the cluster state (etcd snapshot)
restore <cluster> to restore the cluster state from backups
start-aio to quickly setup an all-in-one cluster with 'default' settings
Cluster ops:
add-etcd <cluster> <ip> to add a etcd-node to the etcd cluster
add-master <cluster> <ip> to add a master node to the k8s cluster
add-node <cluster> <ip> to add a work node to the k8s cluster
del-etcd <cluster> <ip> to delete a etcd-node from the etcd cluster
del-master <cluster> <ip> to delete a master node from the k8s cluster
del-node <cluster> <ip> to delete a work node from the k8s cluster
Extra operation:
kcfg-adm <cluster> <args> to manage client kubeconfig of the k8s cluster
Use "ezctl help <command>" for more information about a given command.
部署k8s集群
环境初始化
root@k8s-ansible:/etc/kubeasz# ./ezctl help setup
Usage: ezctl setup <cluster> <step>
available steps:
01 prepare to prepare CA/certs & kubeconfig & other system settings
02 etcd to setup the etcd cluster
03 container-runtime to setup the container runtime(docker or containerd)
04 kube-master to setup the master nodes
05 kube-node to setup the worker nodes
06 network to setup the network plugin
07 cluster-addon to setup other useful plugins
90 all to run 01~07 all at once
10 ex-lb to install external loadbalance for accessing k8s from outside
11 harbor to install a new harbor server or to integrate with an existed one
examples: ./ezctl setup test-k8s 01 (or ./ezctl setup test-k8s prepare)
./ezctl setup test-k8s 02 (or ./ezctl setup test-k8s etcd)
./ezctl setup test-k8s all
./ezctl setup test-k8s 04 -t restart_master
root@k8s-ansible:/etc/kubeasz# ./ezctl setup k8s-01 01
部署etcd cluster
root@k8s-ansible:/etc/kubeasz# ./ezctl setup k8s-01 02
root@k8s-master1:~# export NODE_IPS="10.4.7.134 10.4.7.135 10.4.7.136"
root@k8s-master1:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; done
https://10.4.7.134:2379 is healthy: successfully committed proposal: took = 7.187429ms
https://10.4.7.135:2379 is healthy: successfully committed proposal: took = 9.889493ms
https://10.4.7.136:2379 is healthy: successfully committed proposal: took = 9.548355ms
master节点及node节点部署docker:
配置harbor客户端证书同步：
root@k8s-ansible:/etc/kubeasz# vim roles/docker/tasks/main.yml
37 - "/etc/docker/certs.d/www.awen.com"
harbor公钥
root@k8s-harbor:/apps/crest# scp awen.com.crt 10.4.7.133:/etc/kubeasz/roles/docker/template
root@k8s-harbor:/apps/crest# scp awen.com.crt 10.4.7.133:/etc/kubeasz/roles/docker/files
root@k8s-harbor: scp /apps/harbor/certs/awen.com.crt root@10.4.7.131:/etc/docker/certs.d/www.awen.com/
root@k8s-ansible:/etc/kubeasz# vim roles/docker/tasks/main.yml
80 - name: docker证书分发
81 copy: src=awen.com.crt dest=/etc/docker/certs.d/www.awen.com/ mode=0644
配置harbor域名解析:
83 - name: 添加域名解析
84 shell: "echo '10.4.7.131 www.awen.com' >> /etc/hosts"
自动分发-harbor客户端证书到各master及node节点
root@k8s-ansible:/etc/kubeasz# ./ezctl setup k8s-01 03
root@k8s-master1:~# docker login www.awen.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
部署master节点：
root@k8s-ansible:/etc/kubeasz# docker pull registry.cn-hangzhou.aliyuncs.com/zhangshijie/pause:3.9
root@k8s-ansible:/etc/kubeasz# docker tag registry.cn-hangzhou.aliyuncs.com/zhangshijie/pause:3.9 www.awen.com/baseimages/pause:3.9
root@k8s-master1:~# docker push www.awen.com/baseimages/pause:3.9
root@k8s-ansible:/etc/kubeasz# vi clusters/k8s-01/config.yml
50 # [containerd]基础容器镜像
51 SANDBOX_IMAGE: "www.awen.com/baseimages/pause:3.9"
root@k8s-ansible:/etc/kubeasz# vi roles/kube-node/templates/kubelet.service.j2
42 --pod-infra-container-image={{ SANDBOX_IMAGE }} \ #这个service文件会同步每个，master node
改为
42 --pod-infra-container-image=www.awen.com/baseimages/pause:3.9 \
node节点下载镜像不需要登录
部署master节点
root@k8s-ansible:/etc/kubeasz# ./ezctl setup k8s-01 04
root@k8s-ansible:/etc/kubeasz# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.4.7.134 Ready,SchedulingDisabled master 23s v1.23.1
10.4.7.135 Ready,SchedulingDisabled master 23s v1.23.1
部署node节点
root@k8s-ansible:/etc/kubeasz# ./ezctl setup k8s-01 05
root@k8s-ansible:/etc/kubeasz# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.4.7.134 Ready,SchedulingDisabled master 104s v1.23.1
10.4.7.135 Ready,SchedulingDisabled master 104s v1.23.1
10.4.7.137 Ready node 19s v1.23.1
10.4.7.138 Ready node 19s v1.23.1
部署网络
root@k8s-ansible:/etc/kubeasz# ./ezctl setup k8s-01 06
root@k8s-master1:~# calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 10.4.7.135 | node-to-node mesh | up | 12:38:04 | Established |
| 10.4.7.137 | node-to-node mesh | up | 12:38:04 | Established |
| 10.4.7.138 | node-to-node mesh | up | 12:38:04 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-master1:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.4.7.254 0.0.0.0 UG 0 0 0 eth0
10.4.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.200.118.0 10.4.7.137 255.255.255.192 UG 0 0 0 tunl0
10.200.138.192 10.4.7.138 255.255.255.192 UG 0 0 0 tunl0
10.200.182.0 0.0.0.0 255.255.255.192 U 0 0 0 *
10.200.192.0 10.4.7.135 255.255.255.192 UG 0 0 0 tunl0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
添加master节点
root@k8s-ansible:/etc/kubeasz# ./ezctl --help
Usage: ezctl COMMAND [args]
-------------------------------------------------------------------------------------
Cluster setups:
list to list all of the managed clusters
checkout <cluster> to switch default kubeconfig of the cluster
new <cluster> to start a new k8s deploy with name 'cluster'
setup <cluster> <step> to setup a cluster, also supporting a step-by-step way
start <cluster> to start all of the k8s services stopped by 'ezctl stop'
stop <cluster> to stop all of the k8s services temporarily
upgrade <cluster> to upgrade the k8s cluster
destroy <cluster> to destroy the k8s cluster
backup <cluster> to backup the cluster state (etcd snapshot)
restore <cluster> to restore the cluster state from backups
start-aio to quickly setup an all-in-one cluster with 'default' settings
Cluster ops:
add-etcd <cluster> <ip> to add a etcd-node to the etcd cluster
add-master <cluster> <ip> to add a master node to the k8s cluster
add-node <cluster> <ip> to add a work node to the k8s cluster
del-etcd <cluster> <ip> to delete a etcd-node from the etcd cluster
del-master <cluster> <ip> to delete a master node from the k8s cluster
del-node <cluster> <ip> to delete a work node from the k8s cluster
Extra operation:
kcfg-adm <cluster> <args> to manage client kubeconfig of the k8s cluster
Use "ezctl help <command>" for more information about a given command.
root@k8s-ansible:/etc/kubeasz# ./ezctl add-master k8s-01 10.4.7.136
root@k8s-ansible:/etc/kubeasz# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.4.7.134 Ready,SchedulingDisabled master 17m v1.23.1
10.4.7.135 Ready,SchedulingDisabled master 17m v1.23.1
10.4.7.136 Ready,SchedulingDisabled master 3m v1.23.1
10.4.7.137 Ready node 15m v1.23.1
10.4.7.138 Ready node 15m v1.23.1
添加node节点
root@k8s-ansible:/etc/kubeasz# ./ezctl add-node k8s-01 10.4.7.139
root@k8s-master1:~# kubectl get node
NAME STATUS ROLES AGE VERSION
10.4.7.134 Ready,SchedulingDisabled master 34m v1.23.1
10.4.7.135 Ready,SchedulingDisabled master 34m v1.23.1
10.4.7.136 Ready,SchedulingDisabled master 20m v1.23.1
10.4.7.137 Ready node 33m v1.23.1
10.4.7.138 Ready node 33m v1.23.1
10.4.7.139 Ready none 62s v1.23.1
root@k8s-master1:~# kubectl apply -f coredns-v1.8.6.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
root@k8s-master2:/etc/docker# docker pull kubernetesui/dashboard:v2.5.1
v2.5.1: Pulling from kubernetesui/dashboard
d1d01ae59b08: Pull complete
a25bff2a339f: Pull complete
Digest: sha256:cc746e7a0b1eec0db01cbabbb6386b23d7af97e79fa9e36bb883a95b7eb96fe2
Status: Downloaded newer image for kubernetesui/dashboard:v2.5.1
docker.io/kubernetesui/dashboard:v2.5.1
root@k8s-master2:/etc/docker# docker tag kubernetesui/dashboard:v2.5.1 www.awen.com/baseimages/dashboard:v2.5.1
root@k8s-master2:/etc/docker# docker push www.awen.com/baseimages/dashboard:v2.5.1
e98b3744f758: Pushed
dab46c9f5775: Pushed
v2.5.1: digest: sha256:0c82e96241aa683fe2f8fbdf43530e22863ac8bfaddb0d7d30b4e3a639d4e8c5 size: 736
root@k8s-master2:/etc/docker# docker pull kubernetesui/metrics-scraper:v1.0.8
v1.0.8: Pulling from kubernetesui/metrics-scraper
978be80e3ee3: Pull complete
5866d2c04d96: Pull complete
Digest: sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c
Status: Downloaded newer image for kubernetesui/metrics-scraper:v1.0.8
docker.io/kubernetesui/metrics-scraper:v1.0.8
root@k8s-master2:/etc/docker# docker tag kubernetesui/metrics-scraper:v1.0.8 www.awen.com/baseimages/metrics-scraper:v1.0.8
root@k8s-master2:/etc/docker# docker push www.awen.com/baseimages/metrics-scraper:v1.0.8
The push refers to repository [www.awen.com/baseimages/metrics-scraper]
bcec7eb9e567: Pushed
d01384fea991: Pushed
v1.0.8: digest: sha256:43227e8286fd379ee0415a5e2156a9439c4056807e3caa38e1dd413b0644807a size: 736
root@k8s-master2:/etc/docker# docker tag kubernetesui/metrics-scraper:v1.0.7 www.awen.com/baseimages/metrics-scraper:v1.0.7
root@k8s-master2:/etc/docker# docker push www.awen.com/baseimages/metrics-scraper:v1.0.7
The push refers to repository [www.awen.com/baseimages/metrics-scraper]
7813555162f0: Pushed
10d0ebb29e3a: Pushed
v1.0.7: digest: sha256:76eb73afa0198ac457c760887ed7ebfa2f58adc09205bd9667b3f76652077a71 size: 736
root@k8s-master1:~# kubectl apply -f dashboard-v2.5.1.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
root@k8s-master1:~# kubectl get secret -n kubernetes-dashboard
NAME TYPE DATA AGE
default-token-2tpc6 kubernetes.io/service-account-token 3 2m14s
kubernetes-dashboard-certs Opaque 0 2m14s
kubernetes-dashboard-csrf Opaque 1 2m14s
kubernetes-dashboard-key-holder Opaque 2 2m14s
kubernetes-dashboard-token-vclqk kubernetes.io/service-account-token 3 2m14s
查看token
root@k8s-master1:~# kubectl describe secret -n kubernetes-dashboard
token: eyJhbGciOiJSUzI1NiIsImtpZCI6InNSU2ktek9tbk1Zcks2bFhqdFZtSG5FNFpsOW5KS2UxZXlHVEhmWkM3NzQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi12Y2xxayIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE5Y2E3OTQxLTUwMjMtNDMwYy1hZjEyLWFhYjI1ZDFjYzNhOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.gr_NA0ptRQ9aUtyvHae59DMyRrve3ogta46N9g3vsjIcmA6mcwO8Lij670pmhpd0bIBeBUtLGUrb_K14TqAJL4SJCqvscyaWNuS2hdoBdz5WpZg2FehXlt2SULmXADBgrrxlKorge_bImWj5ofzoXzMfrxsvVCNMO3ZovKNnbSVNLjgFOnQip4U1j6hoXUAETmxTnuppFkKV6y8AjpyySiUr6HppB_9xBQAct5YTe0n5vKJmhukNjToUC1ywfPaGY4kSderlujsPaWcepz9D4VLfzYLm2vxKDh0R0Hid6EDJbpJ1a30pkAtRc5JEe8sSQDdoUxuWLReVonJc48_Tyw
创建测试容器
root@k8s-master1:~# kubectl run net-test1 --image=alpine sleep 10000000
pod/net-test1 created
root@k8s-master1:~# kubectl run net-test2 --image=alpine sleep 10000000
pod/net-test2 created
root@k8s-master1:~# kubectl run net-test3 --image=alpine sleep 10000000
pod/net-test3 created
kubectl run busybox --image busybox:1.28 -it -- sh
root@k8s-master1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
net-test1 1/1 Running 0 42s
net-test2 1/1 Running 0 37s
net-test3 1/1 Running 0 32s
root@k8s-master1:~# kubectl exec -it net-test1 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # ping www.baidu.com
PING www.baidu.com (14.119.104.189): 56 data bytes
64 bytes from 14.119.104.189: seq=0 ttl=127 time=39.238 ms
64 bytes from 14.119.104.189: seq=1 ttl=127 time=38.084 ms
64 bytes from 14.119.104.189: seq=2 ttl=127 time=40.290 ms
^C
--- www.baidu.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 38.084/39.204/40.290 ms
docker.io/calico/cni:v3.19.3
root@k8s-master1:~# docker tag calico/cni:v3.19.3 www.awen.com/baseimages/calico-cni:v3.19.3
root@k8s-master1:~# docker push www.awen.com/baseimages/calico-cni:v3.19.3
The push refers to repository [www.awen.com/baseimages/calico-cni]
c29f7d51e04b: Pushed
7c1fe35a6282: Pushed
11721ef9a0e9: Pushed
v3.19.3: digest: sha256:ac122ab20139a950309bd28f2fa5511d69c9ee337a1fb21534f52f38153215ad size: 946
v3.19.3: Pulling from calico/pod2daemon-flexvol
Digest: sha256:ed08428f8cbf68192c2934fa31db4ae67a6cd1aa47481159e84e509fd5188f81
Status: Image is up to date for calico/pod2daemon-flexvol:v3.19.3
docker.io/calico/pod2daemon-flexvol:v3.19.3
root@k8s-master1:~# docker tag docker.io/calico/pod2daemon-flexvol:v3.19.3 www.awen.com/baseimages/calico-pod2daemon-flexvol:v3.19.3
root@k8s-master1:~# docker push www.awen.com/baseimages/calico-pod2daemon-flexvol:v3.19.3
The push refers to repository [www.awen.com/baseimages/calico-pod2daemon-flexvol]
fb3f05f119d6: Pushed
e0b924e3cec0: Pushed
3b3401d7b4af: Pushed
bfe49c55aaee: Pushed
90d2c07f233d: Pushed
ef32a8de905f: Pushed
72c95e9735b7: Pushed
v3.19.3: digest: sha256:24d28d47d36823669a2fbd9f8f0135f27f15b393dcf771d533c28897264bedd9 size: 1788
root@k8s-master1:~# docker pull calico/kube-controllers:v3.19.3
v3.19.3: Pulling from calico/kube-controllers
Digest: sha256:9079351b7d6871bda159f77c2f917c04d6a7a71cb08bcab5171542f1081cb780
Status: Image is up to date for calico/kube-controllers:v3.19.3
docker.io/calico/kube-controllers:v3.19.3
root@k8s-master1:~# docker tag calico/kube-controllers:v3.19.3 www.awen.com/baseimages/calico-kube-controllers:v3.19.3
root@k8s-master1:~# docker push www.awen.com/baseimages/calico-kube-controllers:v3.19.3
The push refers to repository [www.awen.com/baseimages/calico-kube-controllers]
a699cc477a5a: Pushed
9ce8e113ac65: Pushed
dbe141c11604: Pushed
278d839559a1: Pushed
v3.19.3: digest: sha256:64344f7ca801b6043d4439f02ea674c7eb359d30da96f765dcec3d8fd387d196 size: 1155
root@k8s-master1:~# docker pull calico/node:v3.19.3
v3.19.3: Pulling from calico/node
Digest: sha256:a9b54f6fc3a22088c3dddbb1b268971df9003dd6170a6a92f82210e1311d1c41
Status: Image is up to date for calico/node:v3.19.3
docker.io/calico/node:v3.19.3
root@k8s-master1:~# docker tag calico/node:v3.19.3 www.awen.com/baseimages/calico-node:v3.19.3
root@k8s-master1:~# docker push www.awen.com/baseimages/calico-node:v3.19.3
The push refers to repository [www.awen.com/baseimages/calico-node]
5c8b8d7a47a1: Pushed
803fb24398e2: Pushed
v3.19.3: digest: sha256:d84a6c139c86fabc6414267ee4c87879d42d15c2e0eaf96030c91a59e27e7a6f size: 737
浙公网安备 33010602011771号