arm ubuntu安装kubernetes和kubesphere
一、集群规划
master 192.168.107.128 (master)
node1 192.168.107.129 (slave)
node2 192.168.107.130 (slave)二、必要环境准备(3台都执行)
(1) 都关闭防火墙
systemctl stop ufw
systemctl disable ufw(2) 都关闭swap
swapoff -a # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久(3) 根据规划设置主机名
hostnamectl set-hostname master
hostnamectl set-hostname node1
hostnamectl set-hostname node2(4) 只各机器添加hosts
cat >> /etc/hosts << EOF
192.168.215.140 node1
192.168.215.141 node2
192.168.215.142 node3
EOF(5) 三台服务器，将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF
sysctl --system(6) 时间同步
yum install ntpdate -y
ntpdate time.windows.com（7）安装docker
sudo apt update
sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker（8）修改cgroups为systemd
vim /etc/docker/daemon.json
# 将下面的json内容添加到文件中
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
systemctl daemon-reload
systemctl restart docker二、安装kubernetes
（1）添加kubernetes的镜像源
sudo apt-get update && sudo apt-get install -y ca-certificates curl software-properties-common apt-transport-https curl
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
sudo tee /etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update # 这里要更新源，否则没法安装（2）安装相应版本的kubectl，kubelet和kubeadm
sudo apt install kubectl=1.21.0-00 kubelet=1.21.0-00 kubeadm=1.21.0-00
sudo apt-mark hold kubelet kubeadm kubectl（3）查看kubernetes安装所需要的docker镜像
kubeadm config images list --kubernetes-version=v1.21.0
结果：
k8s.gcr.io/kube-apiserver:v1.21.0
k8s.gcr.io/kube-controller-manager:v1.21.0
k8s.gcr.io/kube-scheduler:v1.21.0
k8s.gcr.io/kube-proxy:v1.21.0
k8s.gcr.io/pause:3.4.1
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns/coredns:v1.8.0（4）由于外网的原因，从阿里云下载镜像这里下载的各个镜像后边的版本，是跟上边的版本一一对应的，这个需要注意，自己找好对应的版本
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.21.0
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.21.0
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.21.0
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.21.0
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.4.1
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.0
下载好后改名为kubernetes需要的名字
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.21.0 k8s.gcr.io/kube-apiserver:v1.21.0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.21.0 k8s.gcr.io/kube-controller-manager:v1.21.0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.21.0 k8s.gcr.io/kube-scheduler:v1.21.0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.21.0 k8s.gcr.io/kube-proxy:v1.21.0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.4.1 k8s.gcr.io/pause:3.4.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0 k8s.gcr.io/etcd:3.4.13-0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.0 k8s.gcr.io/coredns/coredns:v1.8.0（5）master节点初始化
kubeadm init \
--kubernetes-version=v1.21.0 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12出现如下内容就是安装成功
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.107.128:6443 --token rjgf90.4glae3d5cq3n72uz \
--discovery-token-ca-cert-hash sha256:8db99c28c7f0ff57b29141d81f74bdf47a95f0441dcb4bb0a734f46fcf67e518（6）根据安装成功后的提示，我们应该执行如下代码
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config三、安装网络插件calico
在初始化的时候我们配置了--pod-network-cidr=10.244.0.0/16 就是给calico用的，安装calico执行如下命令：
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/custom-resources.yaml这里碰到raw.githubusercontent.com域名无法解析问题，使用域名查询网站，找到对应IP加入至hosts文件中后，可正常下载yaml文件
安装完毕后，执行kubectl get pods -n calico-system查看pods如下：
看到ready这里都准备好后即安装成功
故障排除：
kube-proxy容器有两个在node节点上的无法running
查看kube-proxy容器
kubectl describe pod kube-proxy-2r7qr -n kube-system
根据容器详情发现kube-proxy-2r7qr容器的镜像无法拉取.因为两个node节点未事先拉取镜像，在node节点上拉取镜像并改名后，容器正常运行。
kubectl get nodes 显示三个节点均为ready状态，kubernetes和calico网络插件安装成功！
四、安装NFS共享存储，为kubesphere安装必备条件
(1)安装NFS服务：
sudo apt install nfs-kernel-server(2)创建共享目录
sudo mkdir -p /nfs/data(3)编写配置文件：
sudo vi /etc/exports
/nfs/data *(rw,sync,no_subtree_check,no_root_squash)(4)重启nfs服务：
sudo service nfs-kernel-server restart(5)检查配置情况
sudo showmount -e localhost
能正常显示 
为配置成功
（6）安装客户端工具：
sudo apt install nfs-common
查询NFS共享
sudo showmount -e 192.168.107.128（7）创建本地挂载目录并挂载NFS
sudo mkdir -p /nfs/data
sudo mount -t nfs 192.168.107.128:/nfs/data /nfs/data五、安装kubesphere
（1）Kubernetes 集群已配置默认 StorageClass（请使用 kubectl get sc 进行确认）。
kubectl get sc
No resources found(2) 无sc资源不符合要求，需要先创建存储类
vim sc.yaml
## 创建了一个存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-storage
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
archiveOnDelete: "true" ## 删除pv的时候，pv的内容是否要备份
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
# resources:
# limits:
# cpu: 10m
# requests:
# cpu: 10m
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: k8s-sigs.io/nfs-subdir-external-provisioner
- name: NFS_SERVER
value: 192.168.107.128 ## 指定自己nfs服务器地址
- name: NFS_PATH
value: /nfs/data ## nfs服务器共享的目录
volumes:
- name: nfs-client-root
nfs:
server: 192.168.107.128 ## 指定自己nfs服务器地址
path: /nfs/data
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io(3) 执行查看
kubectl apply -f sc.yaml
结果：
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-storage (default) k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 30s（4）安装一个集群监控指标组件
vim metrics-server.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --kubelet-insecure-tls
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/metrics-server:v0.4.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
periodSeconds: 10
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100(5) 下载KubeSphere yaml文件
wget https://github.com/kubesphere/ks-installer/releases/download/v3.3.0/kubesphere-installer.yaml
wget https://github.com/kubesphere/ks-installer/releases/download/v3.3.0/cluster-configuration.yaml（6）修改cluster-configuration
在 cluster-configuration.yaml中指定我们需要开启的功能
参照官网“启用可插拔组件”
https://kubesphere.io/zh/docs/v3.3/pluggable-components/
（7）执行安装
kubectl apply -f kubesphere-installer.yaml
kubectl apply -f cluster-configuration.yaml
查看安装进度
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f（8）查看所有容器是否启动成功
(9) 所有容器启动成功后登录web控制台
地址：192.168.107.128:30880
帐号：admin
密码：P@88w0rd
安装完成！
六、总结及故障排除
（1）kubesphere安装后登录报错 token failed, reason: getaddrinfo EAI_AGAIN ks-apiserver
解决办法：
编辑coredns 配置信息
kubectl -n kube-system edit cm coredns -o yaml
注释掉：
# forward . /etc/resolv.conf {
# max_concurrent 1000
#}
查看kubectl get pods -n kube-system | grep core
删除掉这两个，之后系统会自动重建
kubectl delete pod coredns-6d8c4cb4d-kvtp5 -n kube-system
kubectl delete pod coredns-6d8c4cb4d-tr22x -n kube-system
重建后错误排除
（2）kubesphere安装中多容器无法正常启动
因为arm版kubesphere 的node-expoter等容器，3.2.0及以前版本没有arm容器镜像，安装3.2.0及以前版本的kubesphere后摘取的是x86版本镜像，在arm平台上无法正常运行。所以arm版本kubesphere 必须安装3.3.0或以上版本
浙公网安备 33010602011771号