Kubernetes
kubeadmin方式搭建
配置： master：2核2G node：1核2G node：1核2G
以下操作在所有节点进行
修改hosts文件并修改hostname
cat > /etc/hosts <<EOF 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.88.30 master-01 192.168.88.31 node-01 192.168.88.32 node-02 EOF
hostnamectl set-hostname
master设置免密登录，生成密钥，分发给各节点
安装docker
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo sudo yum makecache fast sudo yum -y install docker-ce sudo service docker start systemctl enable docker sudo mkdir -p /etc/docker sudo tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://4gokpl83.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker
systemctl enable docker
时间同步
systemctl restart chronyd.service
关闭防火墙，selinux，swap
增加k8s阿里源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
yum makecache
内核升级
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm yum --disablerepo=\* --enablerepo=elrepo-kernel repolist yum --disablerepo=\* --enablerepo=elrepo-kernel list kernel* yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt.x86_64 yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64 -y yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt-tools.x86_64 awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg grub2-set-default 0 reboot
安装 IPVS
yum install -y conntrack-tools ipvsadm ipset conntrack libseccomp
加载 IPVS 模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack" for kernel_module in \${ipvs_modules}; do /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1 if [ $? -eq 0 ]; then /sbin/modprobe \${kernel_module} fi done EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
增加k8s转发配置并使其生效
cat > /etc/sysctl.d/k8s.conf << EOF net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp.keepaliv.probes = 3 net.ipv4.tcp_keepalive_intvl = 15 net.ipv4.tcp.max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp.max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 65536 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.top_timestamps = 0 net.core.somaxconn = 16384 EOF
立即生效
sysctl --system
安装并设置kubelet开机自启( kubectl 只需 master 节点安装 )
yum install -y kubelet-1.22.15 kubeadm-1.22.15 kubectl-1.22.15
systemctl enable kubelet.service
打印集群安装所需的镜像以及版本：
kubeadm config images list
启动kubelet
systemctl start kubelet
在master节点进行
拉取k8s所需镜像到master
docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/kube-apiserver:v1.22.15 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/kube-controller-manager:v1.22.15 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/kube-scheduler:v1.22.15 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/kube-proxy:v1.22.15 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/pause:3.5 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/etcd:3.5.0-0 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/coredns:v1.8.4
进行重新tag操作
docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/kube-proxy:v1.22.15 k8s.gcr.io/kube-proxy:v1.22.15 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/kube-scheduler:v1.22.15 k8s.gcr.io/kube-scheduler:v1.22.15 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/kube-apiserver:v1.22.15 k8s.gcr.io/kube-apiserver:v1.22.15 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/kube-controller-manager:v1.22.15 k8s.gcr.io/kube-controller-manager:v1.22.15 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/etcd:3.5.0-0 k8s.gcr.io/etcd:3.5.0-0 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/pause:3.5 k8s.gcr.io/pause:3.5 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/coredns:v1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4
初始化集群
在master节点进行
kubeadm init \--pod-network-cidr=10.244.0.0/16 \ --apiserver-advertise-address=192.168.88.30
重新初始化之前输入命令
kubeadm reset
防止不能使用kubectl命令
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
在node节点进行
增加节点
kubeadm join 192.168.88.30:6443 --token vlw2sc.5sivgw66jjpgfht2 \ --discovery-token-ca-cert-hash sha256:035f93fd1a7b104239de8739a28436b211a9c35ec996bffd791f1eece76487e4
查看节点状态
kubectl get pod -n kube-system
kubectl describe po kube-proxy-hvb6j -n kube-system
拉取相关镜像并重命名
docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/kube-proxy:v1.22.15 docker pull registry.cn-shenzhen.aliyuncs.com/adif0028/pause:3.5 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/kube-proxy:v1.22.15 k8s.gcr.io/kube-proxy:v1.22.15 docker tag registry.cn-shenzhen.aliyuncs.com/adif0028/pause:3.5 k8s.gcr.io/pause:3.5
在master节点运行
配置flannel网络，可以自己拉取相关镜像
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
查看状态
kubectl get pod -n kube-flannel
kubectl describe po kube-flannel-ds-pvktw -n kube-flannel
修改配置为IPVS
kubectl edit configmap kube-proxy -n kube-system ipvs: ... mode: "ipvs"
删除所有proxy
kubectl get pod -n kube-system
kubectl delete pod kube-proxy-grkj6 -n kube-system kubectl delete pod kube-proxy-l92qb -n kube-system kubectl delete pod kube-proxy-znj7q -n kube-system
查看
kubectl logs kube-proxy-2kjxm -n kube-system I1107 12:22:22.446838 1 server_others.go:276] creating dualStackProxier for ipvs.
安装补全命令
yum -y install bash-completion source /usr/share/bash-completion/bash_completion source <(kubectl completion bash) echo "source <(kubectl completion bash)" >> ~/.bashrc
k8s基础
kubectl 常用命令
# kubectl get service --all-namespaces -o wide # kubectl get pods --all-namespaces -o wide # kubectl get nodes --all-namespaces -o wide # kubectl get deployment --all-namespaces # kubectl get deployment -n magedu -o wide #更改显示格式 # kubectl describe pods magedu-tomcat-app1-deployment -n magedu #查看某个资源详细信息 # kubectl create -f tomcat-app1.yaml # kubectl apply -f tomcat-app1.yaml # kubectl delete -f tomcat-app1.yaml # kubectl create -f tomcat-app1.yaml --save-config --record # kubectl apply -f tomcat-app1.yaml --record #推荐命令 # kubectl exec -it magedu-tomcat-app1-deployment-6bccd8f9c7-g76s5 bash -n magedu # kubectl logs magedu-tomcat-app1-deployment-6bccd8f9c7-g76s5 -n magedu # kubectl delete pods magedu-tomcat-app1-deployment-6bccd8f9c7-g76s5 -n magedu
pod
Pod 中可以同时运行多个进程（作为容器运行）协同工作。同一个 Pod 中的容器会自动的分配到同一个 node
上。同一个 Pod 中的容器共享资源、网络环境和依赖，所以它们总是被同时调度。
Label
Label 是 Kubernetes 系统中另外一个核心概念。一个 Label 是一个 key=value 的键值对，其中 key 与 vaue 由用
户自己指定。Label 可以附加到各种资源对象上，例如 Node、Pod、Service、RC 等，一个资源对象可以定义任意
数量的 Label，同一个 Label 也可以被添加到任意数量的资源对象上去，Label 通常在资源对象定义时确定，也可
以在对象创建后动态添加或者删除。
我们可以通过指定的资源对象捆绑一个或多个不同的 Label 来实现多维度的资源分组管理功能，以便于灵活、
方便地进行资源分配、调度、配置、部署等管理工作。例如：部署不同版本的应用到不同的环境中；或者监控和
分析应用（日志记录、监控、告警）等。一些常用等 label 示例如下。
版本标签："release" : "stable" , "release" : "canary"
环境标签："environment" : "dev" , "environment" : "production"
架构标签："tier" : "frontend" , "tier" : "backend" , "tier" : "middleware"
分区标签："partition" : "customerA" , "partition" : "customerB"
质量管控标签："track" : "daily" , "track" : "weekly"
根据标签来查询 Pod
kubectl get pod --show-labels -n kube-system
增加标签
kubectl describe node node-01 kubectl label node node-01 beta.kubernetes.io/nus=true
删除一个label，只需在命令行最后指定label的key名并与一个减号相连即可：
kubectl label nodes k8s-test01 gpu-
修改一个label的值，需要加上--overwrite参数：
kubectl label nodes k8s-test01 gpu=false --overwrite
service
service 是 k8s 中的一个重要概念，主要是提供负载均衡和服务自动发现。它是 k8s 中最核心的资源之一，每
一个 Service 就是我们平常所说的一个“微服务”。在非 k8s 世界中，管理员可以通过在配置文件中指定 IP 地址
或主机名，容许客户端访问，但在 k8s 中这种方式是行不通的。因为 Pod 是有生命周期的，它们可以被创建或
销毁。虽然通过控制器能够动态地创建 Pod，但当 Pod 被分配到某个节点时，K8s 都会为其分配一个 IP 地址，而
该 IP 地址不总是稳定可依赖的。
如上图所示，Kubernetes 的 Service 定义了一个服务的访问入口，前端的应用（Pod）通过这个入口地址访问
其背后的一组由 Pod 副本组成的集群实例，Service 与其后端的 Pod 副本集群之间是通过 Label Selector 来实现关
联的，而 Deployment 则是保证 Service 的服务能力和服务质量始终处于预期的标准。
通过分析，识别并建模系统中的所有服务为微服务，最终我们的系统是由多个提供不同业务能力而彼此独立
的微服务单元所组成，服务之间通过 TCP/IP 进行通信，从而形成了强大而又灵活的弹性网络，拥有强大的分布
式能力、弹性扩展能力、容错能力。
定义 Service
[root@master-01 project]# vim nginx.yml apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: type: NodePort selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30009 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: type: NodePort selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30009 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.0 ports: - containerPort: 80
kubectl apply -f nginx.yaml
[root@master-01 project]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 24h
nginx NodePort 10.109.192.94 <none> 80:30009/TCP 26s
ngnix-service NodePort 10.107.236.182 <none> 80:30008/TCP 5m36s
进入容器
kubectl exec -it nginx-deployment-66b6c48dd5-268n5 -- bash
更新
spec:
containers:
- name: nginx
image: nginx:1.16.0
kubectl set image deployment nginx nginx=1.19.0
常用：
kubectl apply -f
部署 Dashboard
k8s进阶
安装监控组件 metrics-server
[root@master-01 ~]# vim components.yaml apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server rbac.authorization.k8s.io/aggregate-to-admin: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rbac.authorization.k8s.io/aggregate-to-view: "true" name: system:aggregated-metrics-reader rules: - apiGroups: - metrics.k8s.io resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server name: system:metrics-server rules: - apiGroups: - "" resources: - nodes/metrics verbs: - get - apiGroups: - "" resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: system:metrics-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: v1 kind: Service metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: ports: - name: https port: 443 protocol: TCP targetPort: https selector: k8s-app: metrics-server --- apiVersion: apps/v1 kind: Deployment metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: selector: matchLabels: k8s-app: metrics-server strategy: rollingUpdate: maxUnavailable: 0 template: metadata: labels: k8s-app: metrics-server spec: containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-use-node-status-port - --metric-resolution=15s - --kubelet-insecure-tls image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 name: metrics-server ports: - containerPort: 4443 name: https protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readyz port: https scheme: HTTPS initialDelaySeconds: 20 periodSeconds: 10 resources: requests: cpu: 100m memory: 200Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 volumeMounts: - mountPath: /tmp name: tmp-dir nodeSelector: kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: metrics-server volumes: - emptyDir: {} name: tmp-dir --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: labels: k8s-app: metrics-server name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: kube-system version: v1beta1 versionPriority: 100
安装插件
kubectl apply -f components.yaml
测试
[root@master-01 project]# kubectl top pod NAME CPU(cores) MEMORY(bytes) nginx-deployment-66b6c48dd5-268n5 0m 2Mi nginx-deployment-66b6c48dd5-d8d7z 0m 2Mi nginx-deployment-66b6c48dd5-djqdv 0m 2Mi [root@master-01 project]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-01 143m 7% 1123Mi 60% node-01 34m 3% 372Mi 20% node-02 39m 3% 409Mi 22%
安装nginx ingress
[root@master-01 project]# vim ingress.yml apiVersion: v1 kind: Namespace metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx name: ingress-nginx --- apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx namespace: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx namespace: ingress-nginx rules: - apiGroups: - "" resources: - namespaces verbs: - get - apiGroups: - "" resources: - configmaps - pods - secrets - endpoints verbs: - get - list - watch - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - "" resourceNames: - ingress-controller-leader resources: - configmaps verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - coordination.k8s.io resourceNames: - ingress-controller-leader resources: - leases verbs: - get - update - apiGroups: - coordination.k8s.io resources: - leases verbs: - create - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission namespace: ingress-nginx rules: - apiGroups: - "" resources: - secrets verbs: - get - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets - namespaces verbs: - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission rules: - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations verbs: - get - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: v1 data: allow-snippet-annotations: "true" kind: ConfigMap metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-controller namespace: ingress-nginx --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-controller namespace: ingress-nginx spec: ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http - appProtocol: https name: https port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: NodePort --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-controller-admission namespace: ingress-nginx spec: ports: - appProtocol: https name: https-webhook port: 443 targetPort: webhook selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-controller namespace: ingress-nginx spec: minReadySeconds: 0 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx template: metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx spec: containers: - args: - /nginx-ingress-controller - --election-id=ingress-controller-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so image: registry.cn-shenzhen.aliyuncs.com/adif0028/ingress-nginx:v1.4.0 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: controller ports: - containerPort: 80 name: http protocol: TCP - containerPort: 443 name: https protocol: TCP - containerPort: 8443 name: webhook protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 100m memory: 90Mi securityContext: allowPrivilegeEscalation: true capabilities: add: - NET_BIND_SERVICE drop: - ALL runAsUser: 101 volumeMounts: - mountPath: /usr/local/certificates/ name: webhook-cert readOnly: true dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission-create namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission-create spec: containers: - args: - create - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc - --namespace=$(POD_NAMESPACE) - --secret-name=ingress-nginx-admission env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: registry.cn-shenzhen.aliyuncs.com/adif0028/kube-webhook-certgen:v20220916 imagePullPolicy: IfNotPresent name: create securityContext: allowPrivilegeEscalation: false nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccountName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission-patch namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission-patch spec: containers: - args: - patch - --webhook-name=ingress-nginx-admission - --namespace=$(POD_NAMESPACE) - --patch-mutating=false - --secret-name=ingress-nginx-admission - --patch-failure-policy=Fail env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: registry.cn-shenzhen.aliyuncs.com/adif0028/kube-webhook-certgen:v20220916 imagePullPolicy: IfNotPresent name: patch securityContext: allowPrivilegeEscalation: false nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccountName: ingress-nginx-admission --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: nginx spec: controller: k8s.io/ingress-nginx --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.4.0 name: ingress-nginx-admission webhooks: - admissionReviewVersions: - v1 clientConfig: service: name: ingress-nginx-controller-admission namespace: ingress-nginx path: /networking/v1/ingresses failurePolicy: Fail matchPolicy: Equivalent name: validate.nginx.ingress.kubernetes.io rules: - apiGroups: - networking.k8s.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - ingresses sideEffects: None
安装
kubectl apply -f ingress.yml
验证
[root@master-01 project]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-zcrml 0/1 Completed 0 35m ingress-nginx-admission-patch-pfhn2 0/1 Completed 2 35m ingress-nginx-controller-6bc5fcd464-5s8cc 1/1 Running 0 35m
Ingress的基本使用
安装完ingress后起一台ngnix容器
[root@master-01 ~]# vim test.yml apiVersion: v1 kind: Service metadata: name: nginx namespace: test labels: app: nginx spec: type: NodePort selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30007 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: test labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.0 ports: - containerPort: 80 resources: limits: cpu: 500m memory: 500Mi requests: cpu: 500m memory: 500Mi --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx namespace: test annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.test.com http: paths: - path: / pathType: Prefix backend: service: name: nginx port: number: 80
kubectl apply -f test.yml
查看端口
[root@master-01 ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller NodePort 10.105.48.25 <none> 80:31143/TCP,443:31789/TCP 20m ingress-nginx-controller-admission ClusterIP 10.109.244.16 <none> 443/TCP 20m
在节点上安装Nginx并设置4层代理到ingress，ingress再找到svc
nginx四层配置
[root@master-01 ~]# grep -Ev '^#|[.*#]|^\s*$' /usr/local/nginx/conf/nginx.conf worker_processes 1; events { worker_connections 1024; } stream { upstream ingress { } server { listen 80; proxy_pass ingress; } } http { default_type application/octet-stream; sendfile on; keepalive_timeout 65; }
配置hosts
192.168.88.30 www.test.com
测试
服务探针
存活性探测（LivenessProbe）
用于判断容器是否存活，即 Pod 是否为 running 状态，如果 LivenessProbe 探针探测到容器不健康，则 kubelet
将 kill 掉容器，并根据容器的重启策略判断按照那种方式重启，如果一个容器不包含 LivenessProbe 探针，则 Kubelet
认为容器的 LivenessProbe 探针的返回值永远成功。存活性探测支持的方法有三种:ExecAction，TCPSocketAction，HTTPGetAction。
参数详解
failureThreshold：最少连续几次探测失败的次数，满足该次数则认为 fail
initialDelaySeconds：容器启动之后开始进行存活性探测的秒数。不填立即进行
periodSeconds：执行探测的频率（秒）。默认为 10 秒。最小值为 1。
successThreshold：探测失败后，最少连续探测成功多少次才被认定为成功，满足该次数则认为 success。（但是如果是 liveness 则必须是 1。最小值是 1。）
timeoutSeconds：每次执行探测的超时时间，默认 1 秒，最小 1 秒。
Exec
[root@master-01 project]# vim nginx.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 livenessProbe: exec: command: - cat - /tmp/health initialDelaySeconds: 5 timeoutSeconds: 1
TCPSocket
spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 livenessProbe: initialDelaySeconds: 5 timeoutSeconds: 1 tcpSocket: port: 80
HTTPGet
spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 livenessProbe: initialDelaySeconds: 5 timeoutSeconds: 1 httpGet: path: / port: 80 host: 127.0.0.1 scheme: HTTP
就绪性探测
用于判断容器是否正常提供服务，即容器的 Ready 是否为 True，是否可以接收请求，如果 ReadinessProbe
探测失败，则容器的 Ready 将设置为 False，控制器将此 Pod 的 Endpoint 从对应的 service 的 Endpoint 列表中移除，
从此不再将任何请求调度此 Pod 上，直到下次探测成功。（剔除此 pod,不参与接收请求不会将流量转发给此 Pod)
HTTPGet
readinessProbe: httpGet: port: 80 path: /demo.html
Exec
readinessProbe: exec: command: - cat - /usr/share/nginx/html/index.html
TCPSocket
readinessProbe: tcpSocket: port: 80
HPA 自动伸缩 、Ingrees
在生产环境中，总会有一些意想不到的事情发生，比如公司网站流量突然升高，此时之前创建的 Pod 已不足
以撑住所有的访问，而运维人员也不可能 24 小时守着业务服务，这时就可以通过配置 HPA，实现负载过高的情
况下自动扩容 Pod 副本数以分摊高并发的流量，当流量恢复正常后，HPA 会自动缩减 Pod 的数量。HPA 是根据
CPU 的使用率、内存使用率自动扩展 Pod 数量的，所以要使用 HPA 就必须定义 Requests 参数。
Ingress 为 Kubernetes 集群中的服务提供了入口，可以提供负载均衡、SSL 终止和基于名称的虚拟主机，在生
产环境中常用的 Ingress 有 Treafik、Nginx、HAProxy、Istio 等。在 Kubernetesv 1.1 版中添加的 Ingress 用于从集群
外部到集群内部 Service 的 HTTP 和 HTTPS 路由，流量从 Internet 到 Ingress 再到 Services 最后到 Pod 上，通常情
况下，Ingress 部署在所有的 Node 节点上。Ingress 可以配置提供服务外部访问的 URL、负载均衡、终止 SSL，并
提供基于域名的虚拟主机。但 Ingress 不会暴露任意端口或协议。
[root@master-01 project]# vim nginx.yml apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: type: NodePort selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30009 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.16.0 ports: - containerPort: 80 resources: limits: cpu: 10m memory: 50Mi requests: cpu: 10m memory: 50Mi livenessProbe: initialDelaySeconds: 5 timeoutSeconds: 1 tcpSocket: port: 80 readinessProbe: tcpSocket: port: 80 --- kind: HorizontalPodAutoscaler apiVersion: autoscaling/v2beta2 metadata: name: nginx spec: maxReplicas: 10 minReplicas: 1 scaleTargetRef: kind: Deployment name: nginx apiVersion: apps/v1 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 20 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.hgou.com http: paths: - path: / pathType: Prefix backend: service: name: nginx port: number: 80
对ngnix进行压测查看是否正常扩容缩容
yum -y install httpd-tools
查看ip
[root@master-01 ~]# kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28m nginx NodePort 10.100.110.122 <none> 80:30009/TCP 3m32s
对Nginx进行压测
ab -n 1000 -c 100 http://10.100.110.122/
查看是否自动扩容
[root@master-01 ~]# kubectl get po NAME READY STATUS RESTARTS AGE nginx-5b5bf5dfbd-2hzvb 1/1 Running 0 2m14s nginx-5b5bf5dfbd-gl4rt 1/1 Running 0 43s nginx-5b5bf5dfbd-kbcxz 1/1 Running 0 12s nginx-5b5bf5dfbd-kts9j 1/1 Running 0 12s nginx-5b5bf5dfbd-ws5k7 1/1 Running 0 28s nginx-5b5bf5dfbd-z9f59 1/1 Running 0 28s
等待一段时间后查看是否自动缩容
[root@master-01 ~]# kubectl get po NAME READY STATUS RESTARTS AGE nginx-5b5bf5dfbd-gl4rt 1/1 Running 0 7m26s nginx-5b5bf5dfbd-ws5k7 1/1 Running 0 7m11s nginx-5b5bf5dfbd-z9f59 1/1 Running 0 7m11s
StatefulSet创建mysql容器并测试hostPath
可能由于数据持久化，mysql不能启动，建议彻底删除相关mysql初始化数据或更换节点进行创建
删除相关节点的数据
[root@node-02 ~]# rm -rf /data/mysqldata
[root@master-01 ~]# vim mysql.yml apiVersion: v1 kind: Service metadata: name: mysql namespace: dev labels: app: mysql spec: ports: - protocol: TCP port: 3306 targetPort: 3306 selector: app: mysql --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql namespace: dev labels: app: mysql spec: serviceName: "mysql" replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: nodeSelector: node-role.kubernetes.io/db: "true" containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "123456" ports: - containerPort: 3306 volumeMounts: - name: data mountPath: /var/lib/mysql volumes: - name: data hostPath: path: /data/mysqldata
查看数据是否映射成功
[root@node-01 ~]# ls /data/mysqldata/
auto.cnf ca.pem client-key.pem ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys
ca-key.pem client-cert.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem
K8S 配置中心
在生产环境中经常会遇到需要修改配置文件的情况，传统的修改方式不仅会影响到服务的正常运行，而且操
作步骤也很繁琐。为了解决这个问题，kubernetes 项目从 1.2 版本引入了 ConfigMap 功能，用于将应用的配置信
息与程序的分离。这种方式不仅可以实现应用程序被的复用，而且还可以通过不同的配置实现更灵活的功能。在
创建容器时，用户可以将应用程序打包为容器镜像后，通过环境变量或者外接挂载文件的方式进行配置注入。
ConfigMap && Secret 是 K8S 中的针对应用的配置中心，它有效的解决了应用挂载的问题，并且支持加密以及热
更新等功能，可以说是一个 k8s 提供的一件非常好用的功能。
注意：
ConfigMap 并不提供保密或者加密功能。 如果你想存储的数据是机密的，请使用 Secret，
或者使用其他第三方工具来保证你的数据的私密性，而不是用 ConfigMap
部署zentao项目
nginx四层配置
[root@master-01 ~]# grep -Ev '^#|[.*#]|^\s*$' /usr/local/nginx/conf/nginx.conf worker_processes 1; events { worker_connections 1024; } stream { upstream ingress { } server { listen 80; proxy_pass ingress; } } http { default_type application/octet-stream; sendfile on; keepalive_timeout 65; }
创建mysql容器
[root@master-01 ~]# vim mysql.yml apiVersion: v1 kind: Service metadata: name: mysql namespace: dev labels: app: mysql spec: ports: - protocol: TCP port: 3306 targetPort: 3306 selector: app: mysql --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql namespace: dev labels: app: mysql spec: serviceName: "mysql" replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: nodeSelector: node-role.kubernetes.io/db: "true" containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "123456" ports: - containerPort: 3306 volumeMounts: - name: data mountPath: /var/lib/mysql volumes: - name: data hostPath: path: /data/mysqldata
准备好配置文件后编写Dockerfile创建zentao镜像
[root@master-01 ~]# vim Dockerfile FROM registry.cn-shenzhen.aliyuncs.com/adif0028/nginx_php:74v3 COPY zentaopms /data/zentao COPY nginx.conf /usr/local/nginx/conf/nginx.conf COPY php.ini /usr/local/php/etc/php.ini
启动zentao容器
[root@master-01 ~]# vim zentao.yml apiVersion: v1 kind: Service metadata: name: zentao namespace: dev labels: app: zentao spec: selector: app: zentao ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: zentao namespace: dev labels: app: zentao spec: replicas: 1 selector: matchLabels: app: zentao template: metadata: labels: app: zentao spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - zentao topologyKey: "kubernetes.io/hostname" nodeSelector: node-role.kubernetes.io/bus: "true" containers: - name: zentao image: zentao:v2 ports: - containerPort: 80 resources: limits: cpu: 500m memory: 500Mi requests: cpu: 500m memory: 500Mi livenessProbe: initialDelaySeconds: 5 timeoutSeconds: 1 tcpSocket: port: 80 readinessProbe: tcpSocket: port: 80 --- kind: HorizontalPodAutoscaler apiVersion: autoscaling/v2beta2 metadata: name: zentao namespace: dev spec: maxReplicas: 10 minReplicas: 1 scaleTargetRef: kind: Deployment name: zentao apiVersion: apps/v1 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: zentao namespace: dev annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.hgou.com http: paths: - path: / pathType: Prefix backend: service: name: zentao port: number: 80
修改hosts文件
192.168.88.30 www.hgou.com
登录系统
配置数据库失败，发现在节点报错
PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/coredns/coredns:v1.8.4"
手动拉取镜像k8s.gcr.io/coredns/coredns:v1.8.4后登录正常
扩容节点
二进制安装
环境配置
cat > /etc/hosts << EOF 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.88.51 master-01 m1 192.168.88.52 master-02 m2 192.168.88.53 master-03 m3 192.168.88.54 node-01 n1 192.168.88.70 k8s-m-vip vip EOF
在所有节点进行
修改hostname
在master-01节点设置免密并分发给各节点
安装docker
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
sudo yum makecache fast
sudo yum -y install docker-ce
sudo service docker start
systemctl enable docker
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://4gokpl83.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
时间同步
systemctl restart chronyd.service
关闭防火墙，selinux，swap
内核升级
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --disablerepo=\* --enablerepo=elrepo-kernel repolist
yum --disablerepo=\* --enablerepo=elrepo-kernel list kernel*
yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt.x86_64
yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64 -y
yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-lt-tools.x86_64
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
grub2-set-default 0
reboot
安装 IPVS
yum install -y conntrack-tools ipvsadm ipset conntrack libseccomp
加载 IPVS 模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in \${ipvs_modules}; do
/sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
if [ $? -eq 0 ]; then
/sbin/modprobe \${kernel_module}
fi
done
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
增加k8s转发配置并使其生效
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp.keepaliv.probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp.max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp.max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.top_timestamps = 0
net.core.somaxconn = 16384
EOF
立即生效
sysctl --system
集群证书
在master01执行
安装证书生成工具并设置执行权限
chmod a+x cfssljson_1.6.3_linux_amd64 chmod a+x cfssl_1.6.3_linux_amd64
移动到/usr/local/bin
mv cfssl_1.6.3_linux_amd64 /usr/local/bin/cfssl mv cfssljson_1.6.3_linux_amd64 /usr/local/bin/cfssljson
生成根证书
mkdir -p /opt/cert/ca
cat > /opt/cert/ca/ca-config.json <<EOF { "signing": { "default": { "expiry": "876000h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "876000h" } } } } EOF
生成根证书请求文件
cat > /opt/cert/ca/ca-csr.json << EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names":[{ "C": "CN", "ST": "GuangZhou", "L": "GuangZhou" }] } EOF
生成根证书
cd /opt/cert/ca
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
部署ETCD集群
节点规划
192.168.88.51 etcd-01
192.168.88.52 etcd-01
192.168.88.53 etcd-01
创建ETCD集群证书
mkdir -p /opt/cert/etcd cd /opt/cert/etcd
cat > etcd-csr.json << EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.88.51", "192.168.88.52", "192.168.88.53", "192.168.88.54", "192.168.88.70" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ShangHai", "L": "ShangHai" } ] } EOF
cat > etcd-csr.json << EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.88.51", "192.168.88.52", "192.168.88.53", "192.168.88.54", "192.168.88.70" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "GuangZhou", "L": "GuangZhou" } ] } EOF
生成ETCD证书
cfssl gencert -ca=../ca/ca.pem -ca-key=../ca/ca-key.pem -config=../ca/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
分发ETCD证书
for ip in m1 m2 m3;do ssh root@${ip} "mkdir -pv /etc/etcd/ssl" scp ../ca/ca*.pem root@${ip}:/etc/etcd/ssl scp ./etcd*.pem root@${ip}:/etc/etcd/ssl done
部署ETCD
下载ETCD安装包并解压
tar -xf etcd-v3.5.5-linux-amd64.tar.gz
for i in m1 m2 m3 do scp ./etcd-v3.5.5-linux-amd64/etcd* root@$i:/usr/local/bin/ done
注册ETCD服务
在三台master节点上执行
ETCD_NAME=`hostname` INTERNAL_IP=`hostname -i` INITIAL_CLUSTER=master-01=https://192.168.88.51:2380,master-02=https://192.168.88.52:2380,master-03=https://192.168.88.53:2380
cat << EOF | sudo tee /usr/lib/systemd/system/etcd.service [Unit] Description=etcd Documentation=https://github.com/coreos [Service] ExecStart=/usr/local/bin/etcd \\ --name ${ETCD_NAME} \\
--enable-v2 \\ --cert-file=/etc/etcd/ssl/etcd.pem \\ --key-file=/etc/etcd/ssl/etcd-key.pem \\ --peer-cert-file=/etc/etcd/ssl/etcd.pem \\ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\ --trusted-ca-file=/etc/etcd/ssl/ca.pem \\ --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\ --peer-client-cert-auth \\ --client-cert-auth \\ --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\ --advertise-client-urls https://${INTERNAL_IP}:2379 \\ --initial-cluster-token etcd-cluster \\ --initial-cluster ${INITIAL_CLUSTER} \\ --initial-cluster-state new \\ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
启动ETCD服务
systemctl enable --now etcd
测试ETCD服务
ETCDCTL_API=3 etcdctl \ --cacert=/etc/etcd/ssl/etcd.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ --endpoints="https://192.168.88.51:2379,https://192.168.88.52:2379,https://192.168.88.53:2379" \ member list --write-out='table'
部署master节点
集群规划
kube-apiserver、控制器、调度器、flannel、etcd、kubelet、kube-proxy、DNS
创建集群CA证书
在master-01节点进行
mkdir /opt/cert/k8s cd /opt/cert/k8s
cat > ca-config.json << EOF { "signing": { "default": { "expiry": "876000h" }, "profiles": { "kubernetes": { "expiry": "876000h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF
cat > ca-csr.json << EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "GuangZhou", "ST": "GuangZhou" } ] } EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
创建kube-apiserver的证书
mkdir /opt/cert/k8s cd /opt/cert/k8s
cat > server-csr.json << EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.88.51", "192.168.88.52", "192.168.88.53", "192.168.88.54", "192.168.88.70", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "ShangHai", "ST": "ShangHai" } ] } EOF
api-server:
cat > server-csr.json << EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.88.51", "192.168.88.52", "192.168.88.53", "192.168.88.54", "192.168.88.70", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "GuangZhou", "ST": "GuangZhou" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
controller-manager：
cat > kube-controller-manager-csr.json << EOF { "CN": "system:kube-controller-manager", "hosts": [ "127.0.0.1", "192.168.88.51", "192.168.88.52", "192.168.88.53", "192.168.88.54", "192.168.88.70" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "GuangZhou", "ST": "GuangZhou", "O": "system:kube-controller-manager", "OU": "System" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
创建kube-scheduler的证书
cat > kube-scheduler-csr.json << EOF { "CN": "system:kube-scheduler", "hosts": [ "127.0.0.1", "192.168.88.51", "192.168.88.52", "192.168.88.53", "192.168.88.54", "192.168.88.70" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "GuangZhou", "ST": "GuangZhou", "O": "system:kube-scheduler", "OU": "System" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
创建kube-proxy证书
cat > kube-proxy-csr.json << EOF { "CN":"system:kube-proxy", "hosts":[], "key":{ "algo":"rsa", "size":2048 }, "names":[ { "C":"CN", "L":"GuangZhou", "ST":"GuangZhou", "O":"system:kube-proxy", "OU":"System" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
创建集群管理员证书
cat > admin-csr.json << EOF { "CN":"admin", "key":{ "algo":"rsa", "size":2048 }, "names":[ { "C":"CN", "L":"GuangZhou", "ST":"GuangZhou", "O":"system:masters", "OU":"System" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
颁发证书
mkdir -pv /etc/kubernetes/ssl cp -p ./{ca*pem,server*pem,kube-controller-manager*pem,kube-scheduler*.pem,kube-proxy*pem,admin*.pem} /etc/kubernetes/ssl for i in m1 m2 m3;do ssh root@$i "mkdir -pv /etc/kubernetes/ssl" scp /etc/kubernetes/ssl/* root@$i:/etc/kubernetes/ssl done
编写配置文件以及下载安装包
tar -xf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
for i in m1 m2 m3 ;do scp kube-apiserver kube-controller-manager kube-proxy kubectl kubelet kube-scheduler root@$i:/usr/local/bin; done
创建集群配置文件
mkdir /opt/conf cd /opt/conf
export KUBE_APISERVER="https://192.168.88.70:8443"
创建kube-controller-manager.kubeconfig
设置集群参数
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-controller-manager.kubeconfig
设置客户端认证参数
kubectl config set-credentials "kube-controller-manager" \ --client-certificate=/etc/kubernetes/ssl/kube-controller-manager.pem \ --client-key=/etc/kubernetes/ssl/kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=kube-controller-manager.kubeconfig
设置上下文参数（在上下文参数中将集群参数和用户参数关联起来）
kubectl config set-context default \ --cluster=kubernetes \ --user="kube-controller-manager" \ --kubeconfig=kube-controller-manager.kubeconfig
配置默认上下文
kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig
创建kube-scheduler.kubeconfig
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials "kube-scheduler" \ --client-certificate=/etc/kubernetes/ssl/kube-scheduler.pem \ --client-key=/etc/kubernetes/ssl/kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context default \ --cluster=kubernetes \ --user="kube-scheduler" \ --kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig
创建kube-proxy.kubeconfig集群配置文件
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials "kube-proxy" \ --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \ --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \ --cluster=kubernetes \ --user="kube-proxy" \ --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
创建超级管理员的集群配置文件
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=admin.kubeconfig
kubectl config set-credentials "admin" \ --client-certificate=/etc/kubernetes/ssl/admin.pem \ --client-key=/etc/kubernetes/ssl/admin-key.pem \ --embed-certs=true \ --kubeconfig=admin.kubeconfig
kubectl config set-context default \ --cluster=kubernetes \ --user="admin" \ --kubeconfig=admin.kubeconfig
kubectl config use-context default --kubeconfig=admin.kubeconfig
颁发集群配置文件
for i in m1 m2 m3; do ssh root@$i "mkdir -pv /etc/kubernetes/cfg" scp ./*.kubeconfig root@$i:/etc/kubernetes/cfg done
创建集群token
TLS_BOOTSTRAPPING_TOKEN=`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
cat > token.csv << EOF ${TLS_BOOTSTRAPPING_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF
for i in m1 m2 m3;do scp token.csv root@$i:/etc/kubernetes/cfg/ done
安装kube-apiserver
在所有的master节点上执行
KUBE_APISERVER_IP=`hostname -i`
cat > /etc/kubernetes/cfg/kube-apiserver.conf << EOF KUBE_APISERVER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/var/log/kubernetes \\ --advertise-address=${KUBE_APISERVER_IP} \\ --default-not-ready-toleration-seconds=360 \\ --default-unreachable-toleration-seconds=360 \\ --max-mutating-requests-inflight=2000 \\ --max-requests-inflight=4000 \\ --default-watch-cache-size=200 \\ --delete-collection-workers=2 \\ --bind-address=0.0.0.0 \\ --secure-port=6443 \\ --allow-privileged=true \\ --service-cluster-ip-range=10.96.0.0/16 \\ --service-node-port-range=30000-52767 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ --authorization-mode=RBAC,Node \ --enable-bootstrap-token-auth=true \ --token-auth-file=/etc/kubernetes/cfg/token.csv \ --kubelet-client-certificate=/etc/kubernetes/ssl/server.pem \ --kubelet-client-key=/etc/kubernetes/ssl/server-key.pem \ --tls-cert-file=/etc/kubernetes/ssl/server.pem \ --tls-private-key-file=/etc/kubernetes/ssl/server-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca.pem \ --service-account-issuer=https://kubernetes.default.svc \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kubernetes/k8s-audit.log \ --etcd-servers=https://192.168.88.51:2379,https://192.168.88.52:2379,https://192.168.88.53:2379 \\ --etcd-cafile=/etc/etcd/ssl/ca.pem \\ --etcd-certfile=/etc/etcd/ssl/etcd.pem \\ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem" EOF
注册kube-apiserver的服务
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] EnvironmentFile=/etc/kubernetes/cfg/kube-apiserver.conf ExecStart=/usr/local/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure RestartSec=10 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
systemctl daemon-reload && systemctl start kube-apiserver && systemctl enable kube-apiserver
对kube-apiserver做高可用
安装高可用软件
yum install -y keepalived haproxy
修改keepalived配置文件（注意修改ip 和 state ）
mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf_bak cd /etc/keepalived KUBE_APISERVER_IP=`hostname -i`
master-01:
cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_kubernetes { script "/etc/keepalived/check_kubernetes.sh" interval 2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state MASTER interface ens33 mcast_src_ip ${KUBE_APISERVER_IP} virtual_router_id 51 priority 100 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.88.70 } } EOF
master-02:
cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_kubernetes { script "/etc/keepalived/check_kubernetes.sh" interval 2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip ${KUBE_APISERVER_IP} virtual_router_id 51 priority 100 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.88.70 } } EOF
master-03:
cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_kubernetes { script "/etc/keepalived/check_kubernetes.sh" interval 2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip ${KUBE_APISERVER_IP} virtual_router_id 51 priority 100 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.88.70 } } EOF
systemctl enable --now keepalived
修改haproxy配置文件
cat > /etc/haproxy/haproxy.cfg <<EOF global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor listen stats bind *:8006 mode http stats enable stats hide-version stats uri /stats stats refresh 30s stats realm Haproxy\ Statistics stats auth admin:admin frontend k8s-master bind 0.0.0.0:8443 bind 127.0.0.1:8443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server kubernetes-master-01 192.168.88.51:6443 check inter 2000 fall 2 rise 2 weight 100 server kubernetes-master-02 192.168.88.52:6443 check inter 2000 fall 2 rise 2 weight 100 server kubernetes-master-03 192.168.88.53:6443 check inter 2000 fall 2 rise 2 weight 100 EOF
systemctl enable --now haproxy.service
部署TLS
在某一节点进行
master-01
cd /opt/conf/
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kubelet-bootstrap.kubeconfig
注意修改修改为token.csv里面的token
kubectl config set-credentials "kubelet-bootstrap" \ --token=21b92779862e99719938fbf761f4d97a \ --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl config set-context default \ --cluster=kubernetes \ --user="kubelet-bootstrap" \ --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
颁发证书
for i in m1 m2 m3; do scp kubelet-bootstrap.kubeconfig root@$i:/etc/kubernetes/cfg/ done
创建TLS低权限用户
export KUBECONFIG=/etc/kubernetes/cfg/admin.kubeconfig
创建一个低权限用户
kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap
部署contorller-manager
在三台master节点上执行
cat > /etc/kubernetes/cfg/kube-controller-manager.conf << EOF KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/var/log/kubernetes \\ --leader-elect=true \\ --cluster-name=kubernetes \\ --bind-address=127.0.0.1 \\ --allocate-node-cidrs=true \\ --cluster-cidr=10.244.0.0/12 \\ --service-cluster-ip-range=10.96.0.0/16 \\ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\ --root-ca-file=/etc/kubernetes/ssl/ca.pem \\ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\ --kubeconfig=/etc/kubernetes/cfg/kube-controller-manager.kubeconfig \\ --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \\ --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \\ --experimental-cluster-signing-duration=87600h0m0s \\ --controllers=*,bootstrapsigner,tokencleaner \\ --use-service-account-credentials=true \\ --node-monitor-grace-period=10s \\ --alsologtostderr=true" EOF
注册服务
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] EnvironmentFile=/etc/kubernetes/cfg/kube-controller-manager.conf ExecStart=/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
启动
systemctl daemon-reload && systemctl enable --now kube-controller-manager.service
部署kube-scheduler
所有master节点执行
cat > /etc/kubernetes/cfg/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--v=2 \ --kubeconfig=/etc/kubernetes/cfg/kube-scheduler.kubeconfig \ --leader-elect=true \ --bind-address=0.0.0.0" EOF
注册服务
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] EnvironmentFile=/etc/kubernetes/cfg/kube-scheduler.conf ExecStart=/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
启动
systemctl daemon-reload && systemctl enable --now kube-scheduler.service
部署kubelet服务
需要在三台master节点上执行
KUBE_HOSTNAME=`hostname` KUBE_HOSTNAME=`hostname -i`
cat > /etc/kubernetes/cfg/kubelet.conf << EOF KUBELET_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/var/log/kubernetes \\ --hostname-override=${KUBE_HOSTNAME} \\ --container-runtime=docker \\ --kubeconfig=/etc/kubernetes/cfg/kubelet.kubeconfig \\ --bootstrap-kubeconfig=/etc/kubernetes/cfg/kubelet-bootstrap.kubeconfig \\ --config=/etc/kubernetes/cfg/kubelet-config.yml \\ --cert-dir=/etc/kubernetes/ssl \\ --image-pull-progress-deadline=15m \\ --pod-infra-container-image=registry.cn-shenzhen.aliyuncs.com/adifk8s/pause:3.5" EOF
cat > /etc/kubernetes/cfg/kubelet-config.yml << EOF kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: ${KUBE_HOSTNAME} port: 10250 readOnlyPort: 10255 cgroupDriver: systemd #cgroupfs clusterDNS: - 10.96.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110 EOF
注册kubelet的服务
cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=/etc/kubernetes/cfg/kubelet.conf ExecStart=/usr/local/bin/kubelet \$KUBELET_OPTS Restart=on-failure RestartSec=10 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
EOF
启动
systemctl daemon-reload && systemctl enable --now kubelet.service
查看集群状态
export KUBECONFIG=/etc/kubernetes/cfg/admin.kubeconfig
kubectl get cs
部署kube-proxy
所有master节点上执行
cat > /etc/kubernetes/cfg/kube-proxy.conf << EOF KUBE_PROXY_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/var/log/kubernetes \\ --config=/etc/kubernetes/cfg/kube-proxy-config.yml" EOF
KUBE_HOSTNAME=`hostname -i` HOSTNAME=`hostname` cat > /etc/kubernetes/cfg/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: ${KUBE_HOSTNAME} healthzBindAddress: ${KUBE_HOSTNAME}:10256 metricsBindAddress: ${KUBE_HOSTNAME}:10249 clientConnection: burst: 200 kubeconfig: /etc/kubernetes/cfg/kube-proxy.kubeconfig qps: 100 hostnameOverride: ${HOSTNAME} clusterCIDR: 10.96.0.0/16 enableProfiling: true mode: "ipvs" kubeProxyIPTablesConfiguration: masqueradeAll: false kubeProxyIPVSConfiguration: scheduler: rr excludeCIDRs: [] EOF
注册服务
cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/etc/kubernetes/cfg/kube-proxy.conf ExecStart=/usr/local/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure RestartSec=10 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
启动
systemctl daemon-reload && systemctl enable --now kube-proxy.service
加入集群节点
查看集群节点加入请求
[root@master-01 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
node-csr-b-98b6mnSuOjmgAnjJWwiv6do_cCZsdlMDYLGz7A2mw 23m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap <none> Pending
node-csr-d5ezVJDY1Yn3lgldeEO9sBds7OdU0cUrIhcYrxbuajE 23m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap <none> Pending
node-csr-vP-W4BfWo7VMkwMTWTtuFBbVrFRqk6bPu6EWIpZ0IaE 23m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap <none> Pending
批准加入
在master-01进行
kubectl certificate approve `kubectl get csr | grep "Pending" | awk '{print $1}'`
安装网络插件
在master-01进行
tar -xf flannel-v0.20.1-linux-amd64.tar.gz
for i in m1 m2 m3;do scp flanneld mk-docker-opts.sh root@$i:/usr/local/bin/ done
etcdv3版本的key添加
etcdctl \ --cacert=/etc/etcd/ssl/ca.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ --endpoints="https://192.168.88.51:2379,https://192.168.88.52:2379,https://192.168.88.53:2379" \ put /coreos.com/network/config '{"Network":"10.244.0.0/12", "SubnetLen": 21, "Backend": {"Type": "vxlan", "DirectRouting": true}}'
注册flannel服务
在所有master节点进行
cat > /usr/lib/systemd/system/flanneld.service << EOF [Unit] Description=Flanneld address After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/usr/local/bin/flanneld \\ -etcd-cafile=/etc/etcd/ssl/ca.pem \\ -etcd-certfile=/etc/etcd/ssl/etcd.pem \\ -etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\ -etcd-endpoints=https://192.168.88.51:2379,https://192.168.88.52:2379,https://192.168.88.53:2379 \\ -etcd-prefix=/coreos.com/network \\ -ip-masq ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env Restart=always RestartSec=5 StartLimitInterval=0 [Install] WantedBy=multi-user.target RequiredBy=docker.service EOF
修改docker启动文件，让flannel接管docker网络
sed -i '/ExecStart/s/\(.*\)/#\1/' /usr/lib/systemd/system/docker.service sed -i '/ExecReload/a ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock' /usr/lib/systemd/system/docker.service sed -i '/ExecReload/a EnvironmentFile=-/run/flannel/subnet.env' /usr/lib/systemd/system/docker.service
先启动flannel,再启动docker
systemctl daemon-reload && systemctl enable --now flanneld.service && systemctl restart docker
验证集群网络
集群节点互ping对方的flannel网络
ping 10.242.80.0
安装集群DNS
在master-01节点进行
下载DNS安装配置文件包
wget https://github.com/coredns/deployment/archive/refs/heads/master.zip
unzip unzip deployment-master.zip cd deployment-master/kubernetes
执行部署命令
./deploy.sh -i 10.96.0.2 -s | kubectl apply -f -
验证集群DNS
kubectl get pods -n kube-system
验证集群
绑定一下超管用户
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubernetes
验证集群DNS和集群网络成功
kubectl run test -it --rm --image=busybox:1.28.3
/ # nslookup kubernetes
部署Node节点
分发软件包
for i in n1;do scp flanneld mk-docker-opts.sh kubernetes/server/bin/kubelet kubernetes/server/bin/kube-proxy root@$i:/usr/local/bin; done
分发证书
for i in n1; do ssh root@$i "mkdir -pv /etc/kubernetes/ssl"; scp -pr /etc/kubernetes/ssl/{ca*.pem,admin*pem,kube-proxy*pem} root@$i:/etc/kubernetes/ssl; done
for i in n1 ;do ssh root@$i "mkdir -pv /etc/etcd/ssl"; scp /etc/etcd/ssl/* root@$i:/etc/etcd/ssl; done
分发配置文件
for i in n1;do scp /usr/lib/systemd/system/docker.service root@$i:/usr/lib/systemd/system/docker.service; scp /usr/lib/systemd/system/flanneld.service root@$i:/usr/lib/systemd/system/flanneld.service; done
for i in n1;do ssh root@$i "mkdir -pv /etc/kubernetes/cfg"; scp /etc/kubernetes/cfg/kubelet.conf root@$i:/etc/kubernetes/cfg/kubelet.conf; scp /etc/kubernetes/cfg/kubelet-config.yml root@$i:/etc/kubernetes/cfg/kubelet-config.yml; scp /usr/lib/systemd/system/kubelet.service root@$i:/usr/lib/systemd/system/kubelet.service; scp /etc/kubernetes/cfg/kubelet.kubeconfig root@$i:/etc/kubernetes/cfg/kubelet.kubeconfig; scp /etc/kubernetes/cfg/kubelet-bootstrap.kubeconfig root@$i:/etc/kubernetes/cfg/kubelet-bootstrap.kubeconfig; scp /etc/kubernetes/cfg/token.csv root@$i:/etc/kubernetes/cfg/token.csv; done
在node节点进行
修改配置文件kubelet-config.yml和kubelet.conf
启动kubelet
systemctl enable --now kubelet.service
常用监控对象的指标采集
对于集群的监控一般我们需要考虑以下几个方面：
-
内部系统组件的状态：比如 kube-apiserver、kube-scheduler、kube-controller-manager、kubedns/coredns 等组件的详细运行状态
-
Kubernetes 节点的监控：比如节点的 cpu、load、disk、memory 等指标
-
业务容器指标的监控（容器CPU、内存、磁盘等）
-
编排级的 metrics：比如 Deployment 的状态、资源请求、调度和 API 延迟等数据指标
监控kube-apiserver
可以通过手动配置如下job来试下对apiserver服务的监控
$ cat prometheus-configmap.yaml ... - job_name: 'kubernetes-apiserver' static_configs: - targets: ['10.96.0.1'] scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
监控集群节点基础指标
分析：
-
-
添加节点的容忍配置
-
挂载宿主机中的系统文件信息
DaemonSet确保集群中每个（部分）node运行一份pod副本，如果删除DaemonSet，其创建的所有pod也被删除，DaemonSet中的pod覆盖整个集群。
apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: monitor labels: app: node-exporter spec: selector: matchLabels: app: node-exporter template: metadata: labels: app: node-exporter spec: hostPID: true hostIPC: true hostNetwork: true nodeSelector: kubernetes.io/os: linux containers: - name: node-exporter image: prom/node-exporter:v1.0.1 args: - --web.listen-address=$(HOSTIP):9100 - --path.procfs=/host/proc - --path.sysfs=/host/sys - --path.rootfs=/host/root - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) - --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$ ports: - containerPort: 9100 env: - name: HOSTIP valueFrom: fieldRef: fieldPath: status.hostIP resources: requests: cpu: 150m memory: 180Mi limits: cpu: 150m memory: 180Mi securityContext: runAsNonRoot: true runAsUser: 65534 volumeMounts: - name: proc mountPath: /host/proc - name: sys mountPath: /host/sys - name: root mountPath: /host/root mountPropagation: HostToContainer readOnly: true tolerations: - operator: "Exists" volumes: - name: proc hostPath: path: /proc - name: dev hostPath: path: /dev - name: sys hostPath: path: /sys - name: root hostPath: path: /
创建node-exporter服务
kubectl apply -f node-exporter.yml
重启初始化Prometheus
kubectl delete -f prometheus.yml
问题来了，如何添加到Prometheus的target中？
-
配置一个Service，后端挂载node-exporter的服务，把Service的地址配置到target中
-
带来新的问题，target中无法直观的看到各节点node-exporter的状态
-
-
把每个node-exporter的服务都添加到target列表中
-
带来新的问题，集群节点的增删，都需要手动维护列表
-
target列表维护量随着集群规模增加
Prometheus的服务发现与Relabeling
之前已经给Prometheus配置了RBAC，有读取node的权限，因此Prometheus可以去调用Kubernetes API获取node信息，所以Prometheus通过与 Kubernetes API 集成，提供了内置的服务发现分别是：Node、Service、Pod、Endpoints、Ingress
配置job即可：
- job_name: 'kubernetes-sd-node-exporter' kubernetes_sd_configs: - role: node
重建查看效果
在真正抓取数据前，Prometheus提供了relabeling的能力。怎么理解？
prometheus_notifications_dropped_total{instance="localhost:9090",job="prometheus"}
instance的值其实则取自于__address__
这种发生在采集样本数据之前，对Target实例的标签进行重写的机制在Prometheus被称为Relabeling。
因此，利用relabeling的能力，只需要将__address__替换成node_exporter的服务地址即可。
- job_name: 'kubernetes-sd-node-exporter' kubernetes_sd_configs: - role: node relabel_configs: - source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace
使用cadvisor实现容器指标的采集
深刻错误：ingress 配置 - host: www.grafana.com 不能访问，替换成www.gra.com成功
注意点：
-
-
通过环境变量设置管理员账户密码
-
GF_SECURITY_ADMIN_USER
-
GF_SECURITY_ADMIN_PASSWORD
-
-
通过设置securityContext的方式让grafana进程使用root启动
-
数据挂载到本地
-
apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: monitor spec: selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: volumes: - name: storage hostPath: path: /data/grafana/ nodeSelector: app: prometheus securityContext: runAsUser: 0 containers: - name: grafana image: grafana/grafana:7.1.1 imagePullPolicy: IfNotPresent ports: - containerPort: 3000 name: grafana env: - name: GF_SECURITY_ADMIN_USER value: admin - name: GF_SECURITY_ADMIN_PASSWORD value: admin readinessProbe: failureThreshold: 10 httpGet: path: /api/health port: 3000 scheme: HTTP initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 30 livenessProbe: failureThreshold: 3 httpGet: path: /api/health port: 3000 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 150m memory: 512Mi requests: cpu: 150m memory: 512Mi volumeMounts: - mountPath: /var/lib/grafana name: storage --- apiVersion: v1 kind: Service metadata: name: grafana namespace: monitor spec: type: ClusterIP ports: - port: 3000 selector: app: grafana --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: grafana namespace: monitor spec: rules: - host: grafana.luffy.com http: paths: - path: / backend: serviceName: grafana servicePort: 3000
很多组件已经为了适配Prometheus采集指标，添加了对应的/metrics api，比如
[root@master-01 ~]# kubectl get po -n kube-system -o wide | grep coredns
coredns-78fcd69978-227p6 1/1 Running 1 (10h ago) 5d21h 10.244.1.31 node-01 <none> <none>
coredns-78fcd69978-55n8f 1/1 Running 1 (10h ago) 5d21h 10.244.1.33 node-01 <none> <none>
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: monitor data: prometheus.yml: | global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'coredns' static_configs: - targets: ['10.244.1.46:9153','10.244.1.47:9153'] - job_name: 'kubernetes-apiserver' static_configs: - targets: ['10.96.0.1'] scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token - job_name: 'kubernetes-node-exporter' kubernetes_sd_configs: - role: node relabel_configs: - source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace
vim prometheus-rbac.yml
apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: monitor --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: - "" resources: - nodes - services - endpoints - pods - nodes/proxy verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - configmaps - nodes/metrics verbs: - get - nonResourceURLs: - /metrics verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: monitor
vim prometheus.yml
apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitor labels: app: prometheus spec: selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: serviceAccountName: prometheus nodeSelector: app: prometheus initContainers: - name: "change-permission-of-directory" image: busybox command: ["/bin/sh"] args: ["-c", "chown -R 65534:65534 /prometheus"] securityContext: privileged: true volumeMounts: - mountPath: "/etc/prometheus" name: config-volume - mountPath: "/prometheus" name: data containers: - image: prom/prometheus:v2.19.2 name: prometheus args: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.path=/prometheus" # 指定tsdb数据路径 - "--web.enable-lifecycle" # 支持热更新，直接执行localhost:9090/-/reload立即生效 - "--web.console.libraries=/usr/share/prometheus/console_libraries" - "--web.console.templates=/usr/share/prometheus/consoles" ports: - containerPort: 9090 name: http volumeMounts: - mountPath: "/etc/prometheus" name: config-volume - mountPath: "/prometheus" name: data resources: requests: cpu: 100m memory: 512Mi limits: cpu: 100m memory: 512Mi volumes: - name: data hostPath: path: /data/prometheus/ - configMap: name: prometheus-config name: config-volume --- apiVersion: v1 kind: Service metadata: name: prometheus namespace: monitor labels: app: prometheus spec: selector: app: prometheus type: ClusterIP ports: - name: web port: 9090 targetPort: http --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: prometheus namespace: monitor annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.prometheus.com http: paths: - path: / pathType: Prefix backend: service: name: prometheus port: number: 9090
vim node-exporter.yml
apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: monitor labels: app: node-exporter spec: selector: matchLabels: app: node-exporter template: metadata: labels: app: node-exporter spec: hostPID: true hostIPC: true hostNetwork: true nodeSelector: kubernetes.io/os: linux containers: - name: node-exporter image: prom/node-exporter:v1.0.1 args: - --web.listen-address=$(HOSTIP):9100 - --path.procfs=/host/proc - --path.sysfs=/host/sys - --path.rootfs=/host/root - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) ports: - containerPort: 9100 env: - name: HOSTIP valueFrom: fieldRef: fieldPath: status.hostIP resources: requests: cpu: 150m memory: 180Mi limits: cpu: 150m memory: 180Mi securityContext: runAsNonRoot: true runAsUser: 65534 volumeMounts: - name: proc mountPath: /host/proc - name: sys mountPath: /host/sys - name: root mountPath: /host/root mountPropagation: HostToContainer readOnly: true tolerations: - operator: "Exists" volumes: - name: proc hostPath: path: /proc - name: dev hostPath: path: /dev - name: sys hostPath: path: /sys - name: root hostPath: path: /
vim grafana.yml
apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: monitor spec: selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: volumes: - name: storage hostPath: path: /data/grafana/ nodeSelector: node-role.kubernetes.io/bus: "true" securityContext: runAsUser: 0 containers: - name: grafana image: grafana/grafana:7.1.1 imagePullPolicy: IfNotPresent ports: - containerPort: 3000 name: grafana env: - name: GF_SECURITY_ADMIN_USER value: admin - name: GF_SECURITY_ADMIN_PASSWORD value: admin readinessProbe: failureThreshold: 10 httpGet: path: /api/health port: 3000 scheme: HTTP initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 30 livenessProbe: failureThreshold: 3 httpGet: path: /api/health port: 3000 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 150m memory: 512Mi requests: cpu: 150m memory: 512Mi volumeMounts: - mountPath: /var/lib/grafana name: storage --- apiVersion: v1 kind: Service metadata: name: grafana namespace: monitor spec: type: ClusterIP ports: - port: 3000 selector: app: grafana --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: grafana namespace: monitor annotations: nginx.ingress.kubernetes.io/rewrite-target: / ingressclass.kubernetes.io/is-default-class: "true" spec: ingressClassName: nginx rules: - host: www.gra.com http: paths: - path: / pathType: Prefix backend: service: name: grafana port: number: 3000
按顺序部署上述资源
kubectl create namespace monitor kubectl apply -f prometheus-configmap.yml kubectl apply -f prometheus-rbac.yml kubectl apply -f node-exporter.yml kubectl apply -f prometheus.yml kubectl apply -f grafana.yml
配置grafana data sources
浙公网安备 33010602011771号