docker-compose 搭建 kafka集群

搭建集群

docker-compose up -d --scale kafka=2        本机启动一个有两个节点的 Kafka 集群

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"             # kafka 把9092端口随机映射到主机的端口
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.199.163           #本机ip
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: test:1:1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

输出结果

 chen@macos  ~/docker/kafka01 sudo docker ps                                                                                                                                                   
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
4ede7dc26df7        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   22 minutes ago      Up 22 minutes       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka01_zookeeper_1
2f7c0a71af03        wurstmeister/kafka       "start-kafka.sh"         22 minutes ago      Up 22 minutes       0.0.0.0:32772->9092/tcp                              kafka01_kafka_2
e0b72d9466e0        wurstmeister/kafka       "start-kafka.sh"         22 minutes ago      Up 22 minutes       0.0.0.0:32771->9092/tcp                              kafka01_kafka_1 

注意：docker-compose up -d --scale kafka=2        # 可以进行扩展或缩容                                                                                                    

基础操作

列出主题

容器内部查询主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181          #可以通过容器的名字或者IP访问
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1002	Replicas: 1002	Isr: 1002
容器外部或内部部查询主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh  --describe --zookeeper 192.168.199.163:2181          #这里因为zookeeper映射到了主题的2181端口
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1002	Replicas: 1002	Isr: 1002

创建主题

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh  --create --topic chat --partitions 3 --zookeeper 192.168.199.163:2181 --replication-factor 2
Created topic chat.
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181
Topic:chat	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: chat	Partition: 0	Leader: 1002	Replicas: 1002,1001	Isr: 1002,1001
	Topic: chat	Partition: 1	Leader: 1001	Replicas: 1001,1002	Isr: 1001,1002
	Topic: chat	Partition: 2	Leader: 1002	Replicas: 1002,1001	Isr: 1002,1001
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1002	Replicas: 1002	Isr: 1002

消息消费

生产者－容器里面
bash-4.4# ./kafka-console-producer.sh --broker-list localhost:9092 --topic chat
>da
>

消费者－容器里面
/opt/kafka_2.12-2.2.0/bin # ./kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic chat --from-beginning
da

注意：上面要在宿主机上只要将localhost:9092换为kafka暴露在主机上端口即可