k8s部署efak监控zookeeper和kafka

1.k8s部署

https://www.cnblogs.com/Leonardo-li/p/18648449

2.k8s部署zookeeper

配合efak工具，需要zookeeper开启JXM的sts.yaml （在下边zookeeper部署文档中有体现，部署对应的jxm sts即可）

https://www.cnblogs.com/Leonardo-li/p/18720249

需要zk-service中增加jxm的端口映射
- name: jmx
port: 9999
targetPort: jxm
- name: admin-server
port: 8080 # 新增 AdminServer 端口
targetPort: 8080

apiVersion: v1
kind: Service
metadata:
  name: zk-service
  namespace: zk
  labels:
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/component: zookeeper
spec:
  type: ClusterIP
  sessionAffinity: None
  ports:
    - name: tcp-client
      port: 32181
      targetPort: client
      nodePort: null
    - name: tcp-follower
      port: 32888
      targetPort: follower
    - name: tcp-election
      port: 33888
      targetPort: election
    - name: jmx
      port: 9999
      targetPort: jxm
    - name: admin-server
      port: 8080    # 新增 AdminServer 端口
      targetPort: 8080
  selector:
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/component: zookeeper

kubectl apply -f zk-svc.yaml

需要zookeeper增加允许efak访问的networkpolicy

# network-policy-zookeeper.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-efak-to-zookeeper
  namespace: zk
spec:
  podSelector:
    matchLabels:
      app: zookeeper
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: efak  # 允许 EFAK 命名空间访问
    ports:
    - protocol: TCP
      port: 32181  # 客户端端口
    - protocol: TCP
      port: 9999  # JMX 端口

kubectl apply -f networkpolicy.yaml

3.k8s部署kafka

配合efak工具，需要kafka开启JXM的sts.yaml （在下边kafka部署文档中有体现，部署对应的jxm sts即可）

https://www.cnblogs.com/Leonardo-li/p/18738880

需要kafka增加一个jxm的service

# kafka-jmx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kafka-jmx
  namespace: kafka
spec:
  clusterIP: None  # Headless Service
  ports:
  - port: 9999
    targetPort: 9999
    name: jmx
  selector:
    app: kafka  # 必须与 Kafka StatefulSet 的 Pod 标签一致

kubectl apply -f  ka-jxm-svc.yaml

需要kafka增加允许efak访问的networkpolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-efak-to-kafka-jmx
  namespace: kafka
spec:
  podSelector:
    matchLabels:
      app: kafka  # 修正为 Kafka Pod 的实际标签
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: efak
    ports:
    - protocol: TCP
      port: 9999

kubectl apply -f network-policy.yaml

4.k8s部署efak

EFAK（Eagle For Apache Kafka）是一款专为Apache Kafka设计的开源监控与管理工具，主要用于实时监控Kafka集群的运行状态，帮助运维人员和开发团队快速定位问题并优化性能。

4.1 efak配置文件修改

有时在efak的deployment中，定义变量会不生效，所以想直接从他的容器中拷贝出来system-config.properties配置文件，修改后发布configmap
主要修改内容有

#zookeeper配置
efak.zk.cluster.alias=cluster
cluster.zk.list=zk-service.zk.svc.cluster.local:32181
cluster.zk.jmx.enable=true
cluster.zk.jmx.port=9999
cluster.zk.command.port=8080
#kafka配置
efak.kafka.cluster.alias=cluster
cluster.efak.servers=kafka-nodeport.kafka.svc.cluster.local:30092
cluster.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi
cluster.efak.jmx.port=9999

全量配置文件

system-config.properties

 ######################################
# multi zookeeper & kafka cluster list
# Settings prefixed with 'kafka.eagle.' will be deprecated, use 'efak.' instead
######################################
efak.zk.cluster.alias=cluster
#cluster.zk.list=zookeeper:2181
cluster.zk.list=zk-service.zk.svc.cluster.local:32181
cluster.zk.jmx.enable=true
cluster.zk.jmx.port=9999
cluster.zk.command.port=8080

######################################
# zookeeper enable acl
######################################
cluster.zk.acl.enable=false
cluster.zk.acl.schema=digest
cluster.zk.acl.username=test
cluster.zk.acl.password=test123

######################################
# kraft broker
######################################
efak.kafka.cluster.alias=cluster
#cluster.efak.servers=kafka-0.kafka.svc.cluster.local:30092,kafka-1.kafka.svc.cluster.local:30092,kafka-2.kafka.svc.cluster.local:30092
cluster.efak.servers=kafka-nodeport.kafka.svc.cluster.local:30092
######################################
# broker size online list
######################################
cluster.efak.broker.size=1

######################################
# zk client thread limit
# Zookeeper cluster allows the number of clients to connect to
######################################
kafka.zk.limit.size=25

######################################
# EFAK webui port
######################################
efak.webui.port=8048

######################################
# kafka jmx acl and ssl authenticate
######################################
cluster.efak.jmx.acl=false
cluster.efak.jmx.user=keadmin
cluster.efak.jmx.password=keadmin123
cluster.efak.jmx.ssl=false
cluster.efak.jmx.truststore.location=/Users/dengjie/workspace/ssl/certificates/kafka.truststore
cluster.efak.jmx.truststore.password=ke123456

######################################
# kafka offset storage
######################################
cluster.efak.offset.storage=kafka

# If offset is out of range occurs, enable this property -- Only suitable for kafka sql
efak.sql.fix.error=false

######################################
# kafka jmx uri
######################################
#cluster.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi
#cluster.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s:30099/jmxrmi
#cluster.efak.jmx.uri=kafka-0.kafka.svc.cluster.local:30099,kafka-1.kafka.svc.cluster.local:30099,kafka-2.kafka.svc.cluster.local:30099
#cluster.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://kafka-0.kafka.svc.cluster.local:9999/jmxrmi,service:jmx:rmi:///jndi/rmi://kafka-1.kafka.svc.cluster.local:9999/jmxrmi,service:jmx:rmi:///jndi/rmi://kafka-2.kafka.svc.cluster.local:9999/jmxrmi
# system-config.properties
#cluster.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://kafka-0.kafka.kafka.svc.cluster.local:9999/jmxrmi,service:jmx:rmi:///jndi/rmi://kafka-1.kafka.kafka.svc.cluster.local:9999/jmxrmi,service:jmx:rmi:///jndi/rmi://kafka-2.kafka.kafka.svc.cluster.local:9999/jmxrmi
cluster.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi
cluster.efak.jmx.port=9999
######################################
# kafka metrics, 15 days by default
######################################

# Whether the Kafka performance monitoring diagram is enabled
efak.metrics.charts=false

# Kafka Eagle keeps data for 30 days by default
efak.metrics.retain=30

######################################
# kafka sql topic records max
######################################
efak.sql.topic.records.max=5000
efak.sql.topic.preview.records.max=10
efak.sql.worknode.port=8787
efak.sql.distributed.enable=FALSE
efak.sql.worknode.rpc.timeout=300000
efak.sql.worknode.fetch.threshold=5000
efak.sql.worknode.fetch.timeout=20000
efak.sql.worknode.server.path=/Users/dengjie/workspace/kafka-eagle-plus/kafka-eagle-common/src/main/resources/works

######################################
# delete kafka topic token
# Set to delete the topic token, so that administrators can have the right to delete
######################################
efak.topic.token=keadmin

######################################
# kafka sasl authenticate
######################################
cluster.efak.sasl.enable=false
cluster.efak.sasl.protocol=SASL_PLAINTEXT
cluster.efak.sasl.mechanism=SCRAM-SHA-256
cluster.efak.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="admin-secret";
# If not set, the value can be empty
cluster.efak.sasl.client.id=
# Add kafka cluster cgroups
cluster.efak.sasl.cgroup.enable=false
cluster.efak.sasl.cgroup.topics=kafka_ads01,kafka_ads02

######################################
# kafka jdbc driver address
# Default use sqlite to store data
######################################
efak.driver=org.sqlite.JDBC
# It is important to note that the '/hadoop/kafka-eagle/db' path must exist.
efak.url=jdbc:sqlite:/hadoop/efak/db/ke.db
efak.username=root
efak.password=smartloli

4.2 发布efak cm

kubectl create configmap efak-config \
  --namespace=efak \
  --from-file=system-config.properties

4.3 创建efak deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: efak
  namespace: efak
spec:
  replicas: 1
  selector:  # 添加 selector 字段
    matchLabels:
      app: efak  # 必须与 template 中的 labels 一致
  template:
    metadata:
      labels:
        app: efak  # 此标签需与 selector.matchLabels 匹配
    spec:
      containers:
      - name: efak
        image: 172.16.4.17:8090/ltzx/nickzurich-efak:3.0.1
        ports:
        - containerPort: 8048
        env:
        - name: EFAK_ZK_SERVERS
          value: "zk-service.zk.svc.cluster.local:32181"
        - name: EFAK_KAFKA_SERVERS
          value: "kafka-nodeport.kafka.svc.cluster.local:30092"
        volumeMounts:
        - name: efak-config
          mountPath: "/opt/efak/conf/system-config.properties"
          subPath: system-config.properties
      volumes:
      - name: efak-config
        configMap:
          name: efak-config

---
apiVersion: v1
kind: Service
metadata:
  name: efak-service
  namespace: efak
spec:
  type: NodePort
  ports:
  - port: 8048
    targetPort: 8048
    nodePort: 30448
  selector:
    app: efak  # 必须匹配 Pod 的标签

kubectl apply -f efak.yaml

4.4 网页访问

http://k8smaster-ip:30448
admin/123456

web页面上的配置和展示大家就自己探索吧！！！

posted @ 2025-03-10 17:08 Leonardo-li 阅读(92) 评论(0) 收藏举报

刷新页面返回顶部

Leonardo-li

k8s部署efak监控zookeeper和kafka

1.k8s部署

2.k8s部署zookeeper

3.k8s部署kafka

4.k8s部署efak

公告