Ubuntu 18 Kubernetes集群的安装和部署以及Helm的安装

首先说一下我的环境，我是在windows 10 上面建了一个ubuntu18的虚拟机，同时由于某些原因不受网络限制，所以安装比较顺利。

Install

1.安装并启用 Docker

sudo apt install docker.io
sudo systemctl enable docker
docker --version

2.添加 Kubernetes signing key 和Repository

sudo apt install curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

3.安装Kubeadm

sudo apt install kubeadm
kubeadm version

#常用命令
重启kubelet服务：
systemctl daemon-reload
systemctl restart kubelet
sudo systemctl restart kubelet.service
 
sudo systemctl daemon-reload
sudo systemctl stop kubelet
sudo systemctl enable kubelet
sudo systemctl start kubelet

4.禁用 swapoff

sudo swapoff -a

sudo sed -i '/ swap / s/^/#/' /etc/fstab

以上的指令我只在一台Ubuntu上执行的（如果你有多台计算机，需要在所有的计算机上执行以上指令，我这里是通过拷贝虚拟机来实现的）

5.准备2台虚拟机k8s-master和k8s-node(我这里把上面的计算机命名为 k8s_master ，copy它并命名为k8s_node)

sudo hostnamectl set-hostname k8s-master #在k8s-master 上执行 IP：192.168.255.229
sudo hostnamectl set-hostname k8s-node #k8s-node 上执行 IP：192.168.255.230

Deploy

1.在master上初始化 Kubernetes

使用kubeadm config print init-defaults可以打印集群初始化默认的使用的配置,使用kubeadm默认配置初始化的集群，会在master节点打上node-role.kubernetes.io/master:NoSchedule的污点，阻止master节点接受调度运行工作负载。这里测试环境只有两个节点，所以将这个taint的effect从NoSchedule改为PreferNoSchedule 还有就是修订kubernet版本1.15.2

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.100.7
  bindPort: 6443
nodeRegistration:
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.15.2
networking:
  podSubnet: 10.244.0.0/16

#sudo kubeadm init --pod-network-cidr=192.168.255.229/2
kubeadm init --config kubeadm.yaml #--ignore-preflight-errors=Swap

#sudo kubeadm init --pod-network-cidr=192.168.100.0/2 这里不应该用现有计算机的ip 这里有解决方案 Kubernetes-dashboard pod is crashing again and again

如果执行中遇到detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd"

请参考Container runtimes执行

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries":["192.168.100.20:80"] 
}
EOF
 
mkdir -p /etc/systemd/system/docker.service.d
#我顺便吧docker的私有仓库也加在里面
# Restart docker.
systemctl daemon-reload
systemctl restart docker

如果遇到port 10251 and 10252 are in use 错误请执行 netstat -lnp | grep 1025 然后kill 进程ID

2在master节点上执行如下：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
#检查 master 
kubectl get nodes

3.Deploy a Pod Network & view the status

sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl get pods --all-namespaces
sudo kubectl get nodes

4.add slave node

在k8s-node上执行

kubeadm join 192.168.254.229:6443 --token ewlb93.v0ohocpvncaxgl16 --discovery-token-ca-cert-hash sha256:2522834081168fbe4b5b05854b964e76f1ea8bac6f8fc5e2be21c93c6a27c427

在k8s-master上检查节点信息：

5.安装 Dashboard 插件 &检查状态

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml 
kubectl get deployment kubernetes-dashboard -n kube-system
kubectl get svc kubernetes-dashboard -n kube-system
#kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

为了简单，我这里直接为 Dashboard 赋予 Admin 的权限(否者会禁止访问)，参考https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

启动代理

kubectl proxy 
#或者 如果需要暴露给其他机器，需要指定address:
kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='^*$'

访问http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

不过我在访问这个地址遇到错误如下：

root@k8s-master:~# kubectl get pods --all-namespaces 
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5c98db65d4-97xrf 1/1 Running 0 83m
kube-system coredns-5c98db65d4-vvtfc 1/1 Running 0 83m
kube-system etcd-k8s-master 1/1 Running 0 83m
kube-system kube-apiserver-k8s-master 1/1 Running 0 83m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 83m
kube-system kube-flannel-ds-amd64-gbg49 1/1 Running 1 80m
kube-system kube-flannel-ds-amd64-hmrcp 1/1 Running 0 82m
kube-system kube-proxy-lbp5k 1/1 Running 0 80m
kube-system kube-proxy-szkb8 1/1 Running 0 83m
kube-system kube-scheduler-k8s-master 1/1 Running 0 83m
kube-system kubernetes-dashboard-7d75c474bb-p5nz5 0/1 CrashLoopBackOff 6 10m
root@k8s-master:~# kubectl logs kubernetes-dashboard-7d75c474bb-p5nz5 -n kube-system
2019/08/07 11:34:33 Using in-cluster config to connect to apiserver
2019/08/07 11:34:33 Using service account token for csrf signing
2019/08/07 11:34:33 Starting overwatch
2019/08/07 11:35:03 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
root@k8s-master:~#

后来在https://stackoverflow.com/questions/52273029/kubernetes-dashboard-pod-is-crashing-again-and-again 找到解决方案，

Make sure you understand the difference between Host network, Pod network and Service network. These 3 networks can not overlap. For example --pod-network-cidr=192.168.0.0/16 must notinclude the IP address of your host, change it to 10.0.0.0/16 or something smaller if necessary.

所以初始化该为 sudo kubeadm init --pod-network-cidr=192.168.100.0/2 就okay 了

创建账号

kubectl create serviceaccount dashboard -n default
kubectl create clusterrolebinding dashboard-admin -n default --clusterrole=cluster-admin --serviceaccount=default:dashboard
kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode

Copy the decoded password and login to dashboard.

在Kubernetes的安装过程中会默认安装很多Service账户，分别有不同的访问权限，要找到对应的token，你可以使用下面的方式：

kubectl -n kube-system get secret
kubectl -n kube-system describe secret certificate-controller-token-4xr9x

上面我们使用token来登录，这里我们还是用上面的dashboard账号来生成kubeconfig文件，以后用生成kubeconfig文件来登录

#配置集群信息
kubectl config set-cluster kubernetes --server=192.168.100.11:6443 --kubeconfig=/root/dashbord.conf

#使用token写入集群验证
#kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode #查看token
kubectl config set-credentials dashboard --token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRhc2hib2FyZC10b2tlbi12dDl4OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxZmM1ZDc1ZS0xMjE2LTQwMDgtYThhOS03ZjEwZGQ1NWJjNWEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkYXNoYm9hcmQifQ.Neb5_blRSig6IU5oPtRIQlQhcELaeI8uu7jeiVEdiR3CLiCZYyiI7X6uNsrpGKAkR-OkGM1gOp09-pmxFI6m4lKHYu9S7R1MNigmQrxfZB4RJ-iYZCNp3Rra7mFrluwY_yMbzuZ__XeYShSOiO1VAS2ezWFGk9adgtbiWZkef_NxmYdwEmTAGkmazhatK9SGDWBea-1seoJx-SGFyA9j0gNcWqNrX93ozFmuNtYrPZSwhYkul-q4NHOz4Dp4Ux1C7gZzTIgBySaYZd5tiJIAmZ-6CV-ukmPtFn7tVlNaDkK4K5N6jzyDttlvHZJtWqBR7iWTyamAKAbycm_BmaQR4Q --kubeconfig=/root/dashboard.conf 
配置上下文和当前上下文
kubectl config set-context dashboard@kubernetes --cluster=kubernetes --user=dashboard --kubeconfig=/root/dashboard.conf 
kubectl config use-context dashboard@kubernetes --kubeconfig=/root/dashboard.conf
kubectl config view --kubeconfig=/root/dashboard.conf

[root@master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
#创建一个dashboard的管理用户
[root@master ~]# kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
#将创建的dashboard用户绑定为管理用户
[root@master ~]# kubectl get secrets -n kube-system | grep dashboard
#获取刚刚创建的用户对应的token名称
dashboard-admin-token-88gxw                      kubernetes.io/service-account-token   3      22s
[root@master ~]# kubectl describe secrets -n kube-system dashboard-admin-token-88gxw 
#查看token的详细信息

[root@master ~]# kubectl get secrets -n kube-system | grep dashboard
#查看刚才创建的token
dashboard-admin-token-88gxw                      kubernetes.io/service-account-token   3      22m
[root@master ~]# kubectl describe secrets -n kube-system dashboard-admin-token-88gxw
#查看token的详细信息，会获取token
[root@master ~]# DASH_TOKEN=$(kubectl get secrets -n kube-system dashboard-admin-token-88gxw -o jsonpath={.data.token} | base64 -d)
#将token的信息生成一个变量
[root@master ~]# kubectl config set-cluster kubernets --server=192.168.100.11:6443 --kubeconfig=/root/dashboard.conf 
#将k8s集群的配置信息写入到一个文件中，文件可自定义
[root@master ~]# kubectl config set-credentials dashboard-admin --token=${DASH_TOKEN} --kubeconfig=/root/dashboard.conf 
#将token的信息也写入到文件中（同一个文件）
[root@master ~]# kubectl config set-context dashboard-admin@kubernetes --cluster=kubernetes --user=dashboard-admin --kubeconfig=/root/dashboard.conf 
#将用户信息也写入文件中（同一个文件）
[root@master ~]# kubectl config use-context dashboard-admin@kubernetes --kubeconfig=/root/dashboard.conf 
#将上下文的配置信息也写入文件中（同一个文件）
[root@master ~]# sz /root/dashboard.conf 
#最后将配置信息导入到客户端本地

将/root/dashboard.conf 文件发送到宿主机，浏览器访问时选择Kubeconfig认证，载入该配置文件，点击登陆，即可实现访问，如图：

卸载集群

想要撤销kubeadm做的事，首先要排除节点，并确保在关闭节点之前要清空节点。

在主节点上运行：

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
然后在需要移除的节点上，重置kubeadm的安装状态：

kubeadm reset

重启master节点如果遇到如下错误

The connection to the server 10.2.0.165:6443 was refused - did you specify the right host or port? 解决方案The connection to the server <host>:6443 was refused - did you specify the right host or port?

sudo -i
swapoff -a
exit
strace -eopenat kubectl version

#如果以上方法没有搞定可以执行以下命令
systemctl status docker #查看docker状态
systemctl status kubelet #查看kubelet的状态
netstat -pnlt | grep 6443 #查看6443的端口应该启动起来了
journalctl -xeu kubelet #查看kubelet的日志解决问题

systemctl restart kubelet #最后重启

-------------------------2019-08-20--------------------------

Kubernetes常用组件部署

Helm的安装

Helm由客户端命helm令行工具和服务端tiller组成，Helm的安装十分简单。下载helm命令行工具到master节点node1的/usr/local/bin下，这里下载的2.14.1版本：

curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz
tar -zxvf helm-v2.14.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin/

为了安装服务端tiller，还需要在这台机器上配置好kubectl工具和kubeconfig文件，确保kubectl工具可以在这台机器上访问apiserver且正常使用。这里的node1节点已经配置好了kubectl。

因为Kubernetes APIServer开启了RBAC访问控制，所以需要创建tiller使用的service account: tiller并分配合适的角色给它。详细内容可以查看helm文档中的Role-based Access Control。这里简单起见直接分配cluster-admin这个集群内置的ClusterRole给它。创建helm-rbac.yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

接下来使用helm部署tiller，tiller默认被部署在k8s集群中的kube-system这个namespace下

kubectl create -f helm-rbac.yaml
helm init --service-account tiller --skip-refresh
kubectl get pod -n kube-system -l app=helm
helm version

最后在k8s-master上修改helm chart仓库的地址为azure提供的镜像地址：

helm repo add stable http://mirror.azure.cn/kubernetes/charts

使用Helm部署Nginx Ingress

为了便于将集群中的服务暴露到集群外部，需要使用Ingress。接下来使用Helm将Nginx Ingress部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的边缘节点上，关于Kubernetes边缘节点的高可用相关的内容可以查看之前整理的Bare metal环境下Kubernetes Ingress边缘节点的高可用,Ingress Controller使用hostNetwork。

我们将node1(192.168.100.7)做为边缘节点，打上Label：

root@k8s-master:~/linux-amd64# kubectl label node k8s-master node-role.kubernetes.io/edge= 
node/k8s-master labeled
root@k8s-master:~/linux-amd64# kubectl get node
NAME         STATUS   ROLES         AGE   VERSION
k8s-master   Ready    edge,master   34m   v1.15.2
k8s-node     Ready    <none>        28m   v1.15.2
root@k8s-master:~/linux-amd64#

stable/nginx-ingress chart的值文件ingress-nginx.yaml如下：

controller:
  replicaCount: 1
  hostNetwork: true
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule
defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule

nginx ingress controller的副本数replicaCount为1，将被调度到k8s-master这个边缘节点上。这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。

helm repo update
helm install stable/nginx-ingress -n nginx-ingress --namespace ingress-nginx  -f ingress-nginx.yaml
helm repo add stable http://mirror.azure.cn/kubernetes/charts

如果访问http://192.168.100.7返回default backend，则部署完成。

kubectl get pod -n ingress-nginx -o wide
kubectl -n ingress-nginx exec nginx-ingress-controller-xxxxx -- cat /etc/nginx/nginx.conf

使用Helm部署dashboard

kubernetes-dashboard.yaml

image:
  repository: k8s.gcr.io/kubernetes-dashboard-amd64
  tag: v1.10.1
ingress:
  enabled: true
  hosts: 
    - k8s.frognew.com
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  tls:
    - secretName: frognew-com-tls-secret
      hosts:
      - k8s.frognew.com
nodeSelector:
    node-role.kubernetes.io/edge: ''
tolerations:
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: PreferNoSchedule
rbac:
  clusterAdminRole: true

安装：

helm install stable/kubernetes-dashboard -n kubernetes-dashboard --namespace kube-system -f kubernetes-dashboard.yaml
kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubectl describe -n kube-system secret/kubernetes-dashboard-token-xxx
helm search kubernetes-dashboard
kubectl get pods -n kube-system -o wide