使用docker 和 kubnernetes 部署单节点/多节点 kafka 环境
参考资料
- https://kafka.apachecn.org/documentation.html#configuration
kafka的broker有三个核心配置
- broker.id
- log.dirs
- zookeeper.connect
docker启动单节点kafka环境
启动zookeeper
可配置的环境变量，https://gallery.ecr.aws/bitnami/zookeeper
$ docker run -it --rm --name my-zookeeper -p 2181:2181 -e ALLOW_ANONYMOUS_LOGIN=yes zookeeper:3.7
# 查看节点状态
$ /apache-zookeeper-3.7.0-bin/bin/zkServer.sh status
启动kafka
可配置的环境变量，https://gallery.ecr.aws/bitnami/kafka
$ docker run -it --rm --name my-kafka -p 9092:9092 --link my-zookeeper \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ZOOKEEPER_CONNECT=my-zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes public.ecr.aws/bitnami/kafka:3.2.3
查看和创建topic
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic test --create --partitions 1 --replication-factor 1
测试本地环境
$ kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --from-beginning
$ kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
docker-compose启动单节点kafka
使用docker-composer部署
version: '3.8'
services:
dk-zookeeper:
image: zookeeper:3.7.0
ports:
- 2182:2181
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
dk-kafka-01:
image: public.ecr.aws/bitnami/kafka:3.2.3
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 0
KAFKA_CFG_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092
# KAFKA_CREATE_TOPICS: "test:1:0" //创建test的topic，具备1分区0副本
KAFKA_ZOOKEEPER_CONNECT: dk-zookeeper:2181
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
ALLOW_PLAINTEXT_LISTENER: yes
depends_on:
- dk-zookeeper
docker-compose启动多节点kafka集群
https://blog.csdn.net/qq_39526294/article/details/124293954
创建自定义网络
$ docker network create zookeeper_network
部署zookeeper
创建zookeeper服务
version: '3.8'
networks:
default:
external:
name: zookeeper_network
services:
zoo1:
image: zookeeper:3.7.0
hostname: zoo1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
zoo2:
image: zookeeper:3.7.0
hostname: zoo2
ports:
- 2182:2181
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181
zoo3:
image: zookeeper:3.7.0
hostname: zoo3
ports:
- 2183:2181
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
启动zookeeper
$ docker-compose up
部署kafka
创建kafka服务
version: '3.8'
networks:
default:
external:
name: zookeeper_network
services:
kafka1:
image: public.ecr.aws/bitnami/kafka:3.2.3
restart: unless-stopped
container_name: kafka1
hostname: kafka1
ports:
- "9092:9092"
external_links:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 1
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.31.18.4:9092
KAFKA_ADVERTISED_HOST_NAME: kafka1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181"
ALLOW_PLAINTEXT_LISTENER: yes
kafka2:
image: public.ecr.aws/bitnami/kafka:3.2.3
restart: unless-stopped
container_name: kafka2
hostname: kafka2
ports:
- "9093:9092"
external_links:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 2
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.225:9093 ## 宿主机IP
KAFKA_ADVERTISED_HOST_NAME: kafka2
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181"
ALLOW_PLAINTEXT_LISTENER: yes
kafka3:
image: public.ecr.aws/bitnami/kafka:3.2.3
restart: unless-stopped
container_name: kafka3
hostname: kafka3
ports:
- "9094:9092"
external_links:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 3
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.225:9094 ## 宿主机IP
KAFKA_ADVERTISED_HOST_NAME: kafka3
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181"
ALLOW_PLAINTEXT_LISTENER: yes
kafka-manager:
image: sheepkiller/kafka-manager:latest
restart: unless-stopped
container_name: kafka-manager
hostname: kafka-manager
ports:
- "9000:9000"
links:
- kafka1
- kafka2
- kafka3
external_links:
- zoo1
- zoo2
- zoo3
environment:
ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_BROKERS: kafka1:9092,kafka2:9093,kafka3:9094
启动kafka
$ docker-compose up
查看kafka ui界面
kubernetes启动kafka集群
注意，这里目的仅仅是临时测试，因而回收策略设置为Delete
手动部署
单节点部署
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
spec:
ports:
- name: zookeeper-port
port: 2181
targetPort: 2181
selector:
app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: zookeeper
name: zookeeper
spec:
replicas: 1
selector:
matchLabels:
app: zookeeper
template:
metadata:
labels:
app: zookeeper
spec:
containers:
#- image: wurstmeister/zookeeper
- image: xxxxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/zookeeper:3.7.0
imagePullPolicy: IfNotPresent
name: zookeeper
ports:
- containerPort: 2181
---
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
app: kafka
spec:
type: NodePort
ports:
- port: 9092
name: kafka-port
targetPort: 9092
protocol: TCP
selector:
app: kafka
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: kafka-deployment
spec:
replicas: 1
selector:
matchLabels:
app: kafka
template:
metadata:
labels:
app: kafka
spec:
containers:
- name: kafka
image: public.ecr.aws/bitnami/kafka:3.2.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9092
env:
- name: KAFKA_ADVERTISED_PORT
value: "9092"
- name: KAFKA_ADVERTISED_HOST_NAME
value: "kafka-host"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper-service:2181"
- name: KAFKA_BROKER_ID
value: "1"
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
多节点部署
https://blog.51cto.com/binghe001/5245687
相对来说比较复杂，网络上较多demo
使用operator部署
https://github.com/strimzi/strimzi-kafka-operator
https://github.com/strimzi/strimzi-kafka-operator/blob/0.34.0/examples/kafka/kafka-ephemeral.yaml
一键部署operator
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
创建单节点的zk-kafka环境
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.34.0/examples/kafka/kafka-ephemeral-single.yaml -n kafka
创建多节点临时存储kafka集群
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.34.0/examples/kafka/kafka-ephemeral.yaml -n kafka
yaml文件如下
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.4.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.4"
storage:
type: ephemeral
zookeeper:
replicas: 3
storage:
type: ephemeral
entityOperator:
topicOperator: {}
userOperator: {}
operator的方式还是方便的，删除环境只需要删除crd即可
$ k get pod -n kafka
NAME READY STATUS RESTARTS AGE
my-cluster-entity-operator-578d74f8f9-p9w5c 1/3 Running 0 16s
my-cluster-kafka-0 1/1 Running 0 60s
my-cluster-kafka-1 1/1 Running 0 60s
my-cluster-kafka-2 1/1 Running 0 60s
my-cluster-zookeeper-0 1/1 Running 0 114s
my-cluster-zookeeper-1 1/1 Running 0 114s
my-cluster-zookeeper-2 1/1 Running 0 114s
strimzi-cluster-operator-58967f75c7-8gp4w 1/1 Running 0 7m2s
测试集群
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic test --create --partitions 1 --replication-factor 1
$ kafka-topics --zookeeper zookeeper-service:2181 --list
$ kafka-console-producer --topic test-topic --broker-list localhost:9092
$ kafka-console-consumer --topic test-topic --from-beginning --bootstrap-server localhost:9092
浙公网安备 33010602011771号