k8s部署zookeeper集群

准备yaml文件部署

cat zk-headless.yaml
apiVersion: v1
kind: Service
metadata:
  name: zk-headless
  namespace: zk
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk


cat zk-np.yaml
apiVersion: v1
kind: Service
metadata:
  name: zk-np
  namespace: zk
  labels:
    app: zk
spec:
  type: NodePort
  ports:
  - port: 2181
    targetPort: 2181
    name: client
    nodePort: 32181
  selector:
    app: zk



cat zk-DisruptionBudget.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
  namespace: zk
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1




cat zk-sts.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
  namespace: zk
spec:
  serviceName: zk-headless
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: zk
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: kubernetes-zookeeper
        image: k8s.gcr.io/kubernetes-zookeeper:1.0-3.4.10
        resources:
          requests:
            memory: "1Gi"
            cpu: "0.5"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
          --servers=3 \
          --data_dir=/var/lib/zookeeper/data \
          --data_log_dir=/var/lib/zookeeper/data/log \
          --conf_dir=/opt/zookeeper/conf \
          --client_port=2181 \
          --election_port=3888 \
          --server_port=2888 \
          --tick_time=2000 \
          --init_limit=10 \
          --sync_limit=5 \
          --heap=512M \
          --max_client_cnxns=60 \
          --snap_retain_count=3 \
          --purge_interval=12 \
          --max_session_timeout=40000 \
          --min_session_timeout=4000 \
          --log_level=INFO"
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/zookeeper
        - name: localtime
          mountPath: /etc/localtime
      volumes:
      - name: localtime
        hostPath:
          path: /etc/localtime
          type: ''

  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "nfs-client"
      resources:
        requests:
          storage: 1Gi

注意：
Pod Disruption Budget (pod 中断 预算)，含义其实是 终止pod前 通过labelSelector机制获取正常运行的pod数目的限制，目的是对主动驱逐的保护措施。

• 场景 节点维护或升级时(kubectl drain) 对应用的自动缩容操作(autoscaling down) 由于节点不可用(not ready)导致的Pod驱逐就不能称之为主动
• 特性 PDB指定一个pod集合在一段时间内存活的最小实例数量或者百分比 作用于一组被同一个控制器管理的pod。例如：RC或者statefulapp 使用PodDisruptionBudget控制器本身无法真正保障指定数量或者百分比的pod存活，PodDisruptionBudget控制器只能保证POD主动逃离的情况下业务不中断或者业务SLA不降级
• 场景局限于：主动驱逐 主动驱逐的场景，如果能够保持存活pod数量，将会非常有用。通过使用Pod Disruption Budget 对象，应用可以保证那些主动移除pod的集群操作永远不会同一时间停掉太多pod，导致服务中断或者服务降级。

kubectl drain 操作时遵循PDB对象的设定，如果在该节点上运行了属于统一服务的多个pod，则为了保证最少存活数量，系统会确保每终止一个pod就会在健康的node上启动新的pod后，再继续终止下一个pod容器。

从版本1.7开始可以通过两个参数来配置PodDisruptionBudget：

1. MinAvailable参数：表示最小可用POD数，表示应用POD集群处于运行状态的最小POD数量，或者是运行状态的POD数同总POD数的最小百分比。
2. MaxUnavailable参数：表示最大不可用PO数，表示应用POD集群处于不可用状态的最大POD数，或者是不可用状态的POD数同总POD数的最大百分比。

这里需要注意的是，MinAvailable参数和MaxUnavailable参数是互斥的，也就是说如果使用了其中一个参数，那么就不能使用另外一个参数了。
比如当进行kubectl drain或者POD主动逃离的时候，kubernetes可以通过下面几种情况来判断是否允许：

1. minAvailable设置成了数值5：应用POD集群中最少要有5个健康可用的POD，那么就可以进行操作。
2. minAvailable设置成了百分数30%：应用POD集群中最少要有30%的健康可用POD，那么就可以进行操作。
3. maxUnavailable设置成了数值5：应用POD集群中最多只能有5个不可用POD，才能进行操作。
4. maxUnavailable设置成了百分数30%：应用POD集群中最多只能有30%个不可用POD，才能进行操作。

在极端的情况下，比如将maxUnavailable设置成0，或者设置成100%，那么就表示不能进行kubectl drain操作。同理将minAvailable设置成100%，或者设置成应用POD集群最大副本数，也表示不能进行kubectl drain操作。

这里面需要注意的是，使用PodDisruptionBudget控制器并不能保证任何情况下都对业务POD集群进行约束，PodDisruptionBudget控制器只能保证POD主动逃离的情况下业务不中断或者业务SLA不降级，例如在执行kubectldrain命令时。

验证

kubectl get all -n zk
NAME              READY   STATUS    RESTARTS   AGE
pod/zookeeper-0   1/1     Running   0          2m25s
pod/zookeeper-1   1/1     Running   0          2m3s
pod/zookeeper-2   1/1     Running   0          101s

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/zk-headless   ClusterIP   None             <none>        2888/TCP,3888/TCP   4m41s
service/zk-np         NodePort    10.102.221.204   <none>        2181:32181/TCP      4m7s

NAME                         READY   AGE
statefulset.apps/zookeeper   3/3     2m25s


zkCli.sh -server 100.100.137.211:32181
/usr/local/jdk/bin/java
Connecting to 100.100.137.211:32181
2025-03-10 03:16:49,469 [myid:] - INFO  [main:Environment@98] - Client environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
2025-03-10 03:16:49,475 [myid:] - INFO  [main:Environment@98] - Client environment:host.name=zk-kafka
2025-03-10 03:16:49,475 [myid:] - INFO  [main:Environment@98] - Client environment:java.version=1.8.0_291
2025-03-10 03:16:49,479 [myid:] - INFO  [main:Environment@98] - Client environment:java.vendor=Oracle Corporation
2025-03-10 03:16:49,479 [myid:] - INFO  [main:Environment@98] - Client environment:java.home=/usr/local/jdk/jre
2025-03-10 03:16:49,479 [myid:] - INFO  [main:Environment@98] - Client environment:java.class.path=/usr/local/zookeeper/bin/../zookeeper-server/target/classes:/usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.7.0.jar:/usr/local/zookeeper/bin/../lib/zookeeper-jute-3.7.0.jar:/usr/local/zookeeper/bin/../lib/zookeeper-3.7.0.jar:/usr/local/zookeeper/bin/../lib/snappy-java-1.1.7.7.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.7.30.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.7.30.jar:/usr/local/zookeeper/bin/../lib/simpleclient_servlet-0.9.0.jar:/usr/local/zookeeper/bin/../lib/simpleclient_hotspot-0.9.0.jar:/usr/local/zookeeper/bin/../lib/simpleclient_common-0.9.0.jar:/usr/local/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/usr/local/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-transport-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-resolver-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-handler-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-common-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-codec-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/netty-buffer-4.1.59.Final.jar:/usr/local/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.17.jar:/usr/local/zookeeper/bin/../lib/jline-2.14.6.jar:/usr/local/zookeeper/bin/../lib/jetty-util-ajax-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/jetty-util-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/jetty-servlet-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/jetty-server-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/jetty-security-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/jetty-io-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/jetty-http-9.4.38.v20210224.jar:/usr/local/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/usr/local/zookeeper/bin/../lib/jackson-databind-2.10.5.1.jar:/usr/local/zookeeper/bin/../lib/jackson-core-2.10.5.jar:/usr/local/zookeeper/bin/../lib/jackson-annotations-2.10.5.jar:/usr/local/zookeeper/bin/../lib/commons-cli-1.4.jar:/usr/local/zookeeper/bin/../lib/audience-annotations-0.12.0.jar:/usr/local/zookeeper/bin/../zookeeper-*.jar:/usr/local/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/usr/local/zookeeper/bin/../conf:
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:java.io.tmpdir=/tmp
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:java.compiler=<NA>
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:os.name=Linux
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:os.arch=amd64
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:os.version=3.10.0-957.el7.x86_64
2025-03-10 03:16:49,480 [myid:] - INFO  [main:Environment@98] - Client environment:user.name=root
2025-03-10 03:16:49,481 [myid:] - INFO  [main:Environment@98] - Client environment:user.home=/root
2025-03-10 03:16:49,481 [myid:] - INFO  [main:Environment@98] - Client environment:user.dir=/root
2025-03-10 03:16:49,481 [myid:] - INFO  [main:Environment@98] - Client environment:os.memory.free=50MB
2025-03-10 03:16:49,483 [myid:] - INFO  [main:Environment@98] - Client environment:os.memory.max=228MB
2025-03-10 03:16:49,484 [myid:] - INFO  [main:Environment@98] - Client environment:os.memory.total=57MB
2025-03-10 03:16:49,489 [myid:] - INFO  [main:ZooKeeper@637] - Initiating client connection, connectString=100.100.137.211:32181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@b065c63
2025-03-10 03:16:49,496 [myid:] - INFO  [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2025-03-10 03:16:49,503 [myid:] - INFO  [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes
2025-03-10 03:16:49,522 [myid:] - INFO  [main:ClientCnxn@1726] - zookeeper.request.timeout value is 0. feature enabled=false
Welcome to ZooKeeper!
JLine support is enabled
[zk: 100.100.137.211:32181(CONNECTING) 0] 2025-03-10 03:16:49,965 [myid:100.100.137.211:32181] - INFO  [main-SendThread(100.100.137.211:32181):ClientCnxn$SendThread@1171] - Opening socket connection to server 100.100.137.211/100.100.137.211:32181.
2025-03-10 03:16:49,965 [myid:100.100.137.211:32181] - INFO  [main-SendThread(100.100.137.211:32181):ClientCnxn$SendThread@1173] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2025-03-10 03:16:49,976 [myid:100.100.137.211:32181] - INFO  [main-SendThread(100.100.137.211:32181):ClientCnxn$SendThread@1005] - Socket connection established, initiating session, client: /100.100.137.74:54770, server: 100.100.137.211/100.100.137.211:32181
2025-03-10 03:16:50,089 [myid:100.100.137.211:32181] - INFO  [main-SendThread(100.100.137.211:32181):ClientCnxn$SendThread@1438] - Session establishment complete on server 100.100.137.211/100.100.137.211:32181, session id = 0x1957ee763740000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: 100.100.137.211:32181(CONNECTED) 0]
[zk: 100.100.137.211:32181(CONNECTED) 0]
[zk: 100.100.137.211:32181(CONNECTED) 0]
[zk: 100.100.137.211:32181(CONNECTED) 0] ls /
[zookeeper]
[zk: 100.100.137.211:32181(CONNECTED) 1] create /zqf
Created /zqf
[zk: 100.100.137.211:32181(CONNECTED) 2] quit

WATCHER::

WatchedEvent state:Closed type:None path:null
2025-03-10 03:16:59,588 [myid:] - INFO  [main:ZooKeeper@1232] - Session: 0x1957ee763740000 closed
2025-03-10 03:16:59,588 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@570] - EventThread shut down for session: 0x1957ee763740000
2025-03-10 03:16:59,591 [myid:] - ERROR [main:ServiceUtils@42] - Exiting JVM with code 0