二进制部署kubernetes v1.30.2集群 七、配置k8s核心组件
配置k8s核心组件
配置apiserver（配置在所有master节点）
1、创建systemd管理文件
在k8s-master01节点执行
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server After=network.target [Service] ExecStart=/data/k8s/bin/kube-apiserver \ --v=2 \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --advertise-address=192.168.110.21 \ --service-cluster-ip-range=10.0.0.0/16 \ --service-node-port-range=20000-49999 \ --etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \ --etcd-cafile=/data/etcd/ssl/ca.pem \ --etcd-certfile=/data/etcd/ssl/etcd.pem \ --etcd-keyfile=/data/etcd/ssl/etcd-key.pem \ --client-ca-file=/data/k8s/ssl/ca.pem \ --tls-cert-file=/data/k8s/ssl/apiserver.pem \ --tls-private-key-file=/data/k8s/ssl/apiserver-key.pem \ --kubelet-client-certificate=/data/k8s/ssl/apiserver.pem \ --kubelet-client-key=/data/k8s/ssl/apiserver-key.pem \ --service-account-key-file=/data/k8s/ssl/sa.pub \ --service-account-signing-key-file=/data/k8s/ssl/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --runtime-config=authentication.k8s.io/v1beta1=true \ --requestheader-client-ca-file=/data/k8s/ssl/front-proxy-ca.pem \ --proxy-client-cert-file=/data/k8s/ssl/front-proxy-client.pem \ --proxy-client-key-file=/data/k8s/ssl/front-proxy-client-key.pem \ --requestheader-allowed-names=front-proxy-client \ --enable-bootstrap-token-auth=true \ --audit-log-path=/data/k8s/logs/audit.log \ --audit-log-maxage=30 Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF
在k8s-master02节点执行
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server After=network.target [Service] ExecStart=/data/k8s/bin/kube-apiserver \ --v=2 \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --advertise-address=192.168.110.22 \ --service-cluster-ip-range=10.0.0.0/16 \ --service-node-port-range=20000-49999 \ --etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \ --etcd-cafile=/data/etcd/ssl/ca.pem \ --etcd-certfile=/data/etcd/ssl/etcd.pem \ --etcd-keyfile=/data/etcd/ssl/etcd-key.pem \ --client-ca-file=/data/k8s/ssl/ca.pem \ --tls-cert-file=/data/k8s/ssl/apiserver.pem \ --tls-private-key-file=/data/k8s/ssl/apiserver-key.pem \ --kubelet-client-certificate=/data/k8s/ssl/apiserver.pem \ --kubelet-client-key=/data/k8s/ssl/apiserver-key.pem \ --service-account-key-file=/data/k8s/ssl/sa.pub \ --service-account-signing-key-file=/data/k8s/ssl/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --runtime-config=authentication.k8s.io/v1beta1=true \ --requestheader-client-ca-file=/data/k8s/ssl/front-proxy-ca.pem \ --proxy-client-cert-file=/data/k8s/ssl/front-proxy-client.pem \ --proxy-client-key-file=/data/k8s/ssl/front-proxy-client-key.pem \ --requestheader-allowed-names=front-proxy-client \ --enable-bootstrap-token-auth=true \ --audit-log-path=/data/k8s/logs/audit.log \ --audit-log-maxage=30 Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF
在k8s-master03节点执行
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server After=network.target [Service] ExecStart=/data/k8s/bin/kube-apiserver \ --v=2 \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --advertise-address=192.168.110.23 \ --service-cluster-ip-range=10.0.0.0/16 \ --service-node-port-range=20000-49999 \ --etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \ --etcd-cafile=/data/etcd/ssl/ca.pem \ --etcd-certfile=/data/etcd/ssl/etcd.pem \ --etcd-keyfile=/data/etcd/ssl/etcd-key.pem \ --client-ca-file=/data/k8s/ssl/ca.pem \ --tls-cert-file=/data/k8s/ssl/apiserver.pem \ --tls-private-key-file=/data/k8s/ssl/apiserver-key.pem \ --kubelet-client-certificate=/data/k8s/ssl/apiserver.pem \ --kubelet-client-key=/data/k8s/ssl/apiserver-key.pem \ --service-account-key-file=/data/k8s/ssl/sa.pub \ --service-account-signing-key-file=/data/k8s/ssl/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --runtime-config=authentication.k8s.io/v1beta1=true \ --requestheader-client-ca-file=/data/k8s/ssl/front-proxy-ca.pem \ --proxy-client-cert-file=/data/k8s/ssl/front-proxy-client.pem \ --proxy-client-key-file=/data/k8s/ssl/front-proxy-client-key.pem \ --requestheader-allowed-names=front-proxy-client \ --enable-bootstrap-token-auth=true \ --audit-log-path=/data/k8s/logs/audit.log \ --audit-log-maxage=30 Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF
2、启动apiserver，并设置为开机启动
systemctl start kube-apiserver
systemctl enable kube-apiserver
3、检查apiserver是否启动成功
systemctl status kube-apiserver curl -k https://localhost:6443/healthz
curl --cert /data/k8s/ssl/apiserver.pem --key /data/k8s/ssl/apiserver-key.pem --cacert /data/k8s/ssl/ca.pem https://192.168.110.20:6443/api/v1/namespaces
配置controller-manger（配置在所有master节点）
1、创建controller-manager服务systemd管理文件
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager After=network.target [Service] ExecStart=/data/k8s/bin/kube-controller-manager \ --v=2 \ --bind-address=0.0.0.0 \ --root-ca-file=/data/k8s/ssl/ca.pem \ --cluster-signing-cert-file=/data/k8s/ssl/ca.pem \ --cluster-signing-key-file=/data/k8s/ssl/ca-key.pem \ --service-account-private-key-file=/data/k8s/ssl/sa.key \ --controllers=*,bootstrapsigner,tokencleaner,csrapproving \ --kubeconfig=/data/k8s/conf/controller-manager.kubeconfig \ --leader-elect=true \ --node-monitor-grace-period=30s \ --node-monitor-period=5s \ --allocate-node-cidrs=true \ --service-cluster-ip-range=10.0.0.0/16 \ --cluster-cidr=172.16.0.0/16 \ --node-cidr-mask-size-ipv4=24 \ --requestheader-client-ca-file=/data/k8s/ssl/front-proxy-ca.pem Restart=always RestartSec=5s [Install] WantedBy=multi-user.target EOF
2、增加controller-manager权限（仅在k8s-master01执行）
cat > /data/k8s/conf/rbac-controller-manager.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:rbac-controller-manager labels: kubernetes.io/bootstrapping: rbac-defaults annotations: rbac.authorization.kubernetes.io/autoupdate: "true" rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["create", "update", "delete"] - apiGroups: ["rbac.authorization.k8s.io"] resources: ["clusterrolebindings", "clusterroles"] verbs: ["escalate", "create", "get", "list", "watch", "patch", "update"] - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests"] verbs: ["get", "list", "watch", "delete", "approve", "sign"] - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/approval"] verbs: ["update", "approve"] - apiGroups: ["certificates.k8s.io"] resources: ["signers"] resourceNames: ["kubernetes.io/kube-apiserver-client-kubelet", "kubernetes.io/kubelet-serving"] verbs: ["approve", "sign"] - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/status"] verbs: ["update", "patch"] - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["create", "get", "update"] - apiGroups: ["storage.k8s.io"] resources: ["csinodes"] verbs: ["get"] - apiGroups: ["apps"] resources: ["deployments", "replicasets", "statefulsets", "replicasets/status", "deployments/status", "controllerrevisions", "daemonsets/status", "daemonsets"] verbs: ["create", "get", "list", "watch", "update", "delete", "patch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: [""] resources: ["secrets", "services", "namespaces", "pods", "serviceaccounts","nodes","endpoints"] verbs: ["create", "get", "list", "watch", "update", "delete", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:rbac-controller-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:rbac-controller-manager subjects: - kind: User name: system:kube-controller-manager apiGroup: rbac.authorization.k8s.io EOF kubectl apply -f /data/k8s/conf/rbac-controller-manager.yaml
3、启动contoller-manager，并设置开机启动
systemctl start kube-controller-manager
systemctl enable kube-controller-manager
4、 检查服务是否启动成功
systemctl status kube-controller-manager
curl -k https://localhost:10257/healthz
kubectl get componentstatuses
配置scheduler（配置在所有master节点）
1、创建scheduler服务systemd管理文件
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler After=network.target [Service] ExecStart=/data/k8s/bin/kube-scheduler \ --v=2 \ --bind-address=0.0.0.0 \ --leader-elect=true \ --kubeconfig=/data/k8s/conf/scheduler.kubeconfig Restart=always RestartSec=5s [Install] WantedBy=multi-user.target EOF
2、启动scheduler，并设置开机启动
systemctl start kube-scheduler
systemctl enable kube-scheduler
3、检查服务是否启动成功
systemctl status kube-scheduler curl -k https://localhost:10259/healthz
kubectl get cs
配置kube-proxy（配置在所有节点）
1、创建kube-proxy配置文件。
cat > /data/k8s/conf/kube-proxy.yaml << EOF apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: contentType: application/vnd.kubernetes.protobuf kubeconfig: /data/k8s/conf/kube-proxy.kubeconfig clusterCIDR: 172.16.0.0/16 iptables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30s ipvs: masqueradeAll: true minSyncPeriod: 5s scheduler: "rr" syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 127.0.0.1:10249 mode: "ipvs" EOF
2、创建kube-proxy服务systemd管理文件
cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Kube Proxy After=network.target [Service] ExecStart=/data/k8s/bin/kube-proxy \\ --config=/data/k8s/conf/kube-proxy.yaml \\ --cluster-cidr=172.16.0.0/16 \\ --v=2 Restart=always RestartSec=5s [Install] WantedBy=multi-user.target EOF
3、启动kube-proxy，并设置开机启动
systemctl start kube-proxy
systemctl enable kube-proxy
配置kubelet（配置在所有节点）
1、配置TLS Bootstrapping的自动证书申请（仅在k8s-master01执行）
cat > /data/k8s/conf/auto-approve-rbac.yaml << EOF # 允许启动引导节点创建 CSR
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: create-csrs-for-bootstrapping subjects: - kind: Group name: system:bootstrappers apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:node-bootstrapper apiGroup: rbac.authorization.k8s.io # 批复 "system:bootstrappers" 组的所有 CSR
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: auto-approve-csrs-for-group subjects: - kind: Group name: system:bootstrappers apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient apiGroup: rbac.authorization.k8s.io # 批复 "system:nodes" 组的 CSR 续约请求
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: auto-approve-renewals-for-nodes subjects: - kind: Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient apiGroup: rbac.authorization.k8s.io #创建权限角色 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:kube-apiserver-to-kubelet rules: - apiGroups: [""] resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics verbs: - "*" #绑定权限 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - kind: User name: kube-apiserver EOF kubectl apply -f /data/k8s/conf/bootstrap-token.yaml kubectl apply -f /data/k8s/conf/auto-approve-rbac.yaml
注意：其中允许创建和初始化CSR的配置中，组名需要在kubelet配置Bootstrap Token中auth-extra-groups项中。system：nodes为k8s集群内置组名，绑定权限的用户名称需要与apiserver签名请求文件中CN值对应。
2、创建kubelet配置文件
cat > /data/k8s/conf/kubelet-conf.yml <<EOF apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: 0.0.0.0 port: 10250 readOnlyPort: 10255 rootDir: /data/kubelet authentication: anonymous: enabled: false webhook: enabled: true x509: clientCAFile: /data/k8s/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 10.0.0.2 clusterDomain: cluster.local containerLogMaxFiles: 5 containerLogMaxSize: 200Mi contentType: application/vnd.kubernetes.protobuf enforceNodeAllocatable: - pods evictionHard: imagefs.available: 10% memory.available: 100Mi nodefs.available: 5% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 3m failSwapOn: true healthzBindAddress: 127.0.0.1 healthzPort: 10248 imageGCHighThresholdPercent: 90 imageGCLowThresholdPercent: 85 serializeImagePulls: true staticPodPath: /data/k8s/manifests streamingConnectionIdleTimeout: 1h serverTLSBootstrap: true rotateCertificates: true EOF
3、创建kubelet服务systemd管理文件
cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=network-online.target containerd.service Wants=network-online.target Requires=containerd.service [Service] ExecStart=/data/k8s/bin/kubelet --config=/data/k8s/conf/kubelet-conf.yml --bootstrap-kubeconfig=/data/k8s/conf/bootstrap-kubelet.kubeconfig --kubeconfig=/data/k8s/conf/kubelet.kubeconfig --cert-dir=/data/k8s/ssl [Install] WantedBy=multi-user.target EOF
4、启动kubelet，并设置为开机启动
systemctl start kubelet
systemctl enable kubelet
查看证书自动申请
1、查看csr
kubectl get csr
2、如存在pending状态，则手动批复
kubectl certificate approve `kubectl get csr | grep Pending | awk -F ' ' '{print$1}'`
3、查看证书是否申请成功。
ll /data/k8s/ssl/kubelet*
浙公网安备 33010602011771号