kafka的搭建, kafka_python的使用

转载：Python操作Kafka的通俗总结（kafka-python） - 知乎 (zhihu.com)

1. 下载zookeeper镜像和kafka镜像

docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka

2.启动镜像

vm设置ssh连接：如何使用Xshell连接VMware上的Linux虚拟机 - shireenlee - 博客园 (cnblogs.com)

docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper

docker run -d --name kafka \
-p 9092:9092 \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ZOOKEEPER_CONNECT=192.168.111.4:2181 \ # 192.168.111.4是本机ip地址，这里是虚拟机地址
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.111.4:9092 \ 
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 wurstmeister/kafka

进入kafka

docker exec -it kafka bash
cd /opt/kafka/bin

创建topic

kafka-topics.sh --create --zookeeper zookeeper:2181 \
--replication-factor 1 --partitions 0 --topic test

查看创建的topic

kafka-topics.sh --zookeeper zookeeper:2181 --list

下载kafka-python

pip install kafka-python

 

设置两个生产者：

from kafka import KafkaProducer
from kafka.errors import kafka_errors
import traceback
import json
import time

def producer_demo():
    # 假设生产的消息为键值对（不是一定要键值对），且序列化方式为json
    producer = KafkaProducer(
        bootstrap_servers=['192.168.111.4:9092'],
        key_serializer=lambda k: json.dumps(k).encode(),
        value_serializer=lambda v: json.dumps(v).encode())
    
    for i in range(1,10):
        time.sleep(1)
        future = producer.send(
            'test',
            key='count_num',  # 同一个key值，会被送至同一个分区
            value=str(i),
            partition=0)  
        print("send {}".format(str(i)))
        try:
            future.get(timeout=10) # 监控是否发送成功
        except kafka_errors:  # 发送失败抛出kafka_errors
            traceback.format_exc()
if __name__ == "__main__":
　　producer_demo()

from kafka import KafkaProducer
from kafka.errors import kafka_errors
import traceback
import json
import time


def producer_demo():
    # 假设生产的消息为键值对（不是一定要键值对），且序列化方式为json
    producer = KafkaProducer(
        bootstrap_servers=['192.168.111.4:9092'],
        key_serializer=lambda k: json.dumps(k).encode(),
        value_serializer=lambda v: json.dumps(v).encode())
   
    a = ["a", "b", "c", "d"]
    for i in a:
        time.sleep(1)
        future = producer.send(
            'kafka_demo',
            key='count_num',  # 同一个key值，会被送至同一个分区
            value=str(i),
            partition=0)  
        print("send {}".format(str(i)))
        try:
            future.get(timeout=10)  # 监控是否发送成功
        except kafka_errors:  # 发送失败抛出kafka_errors
            traceback.format_exc()

if __name__ == '__main__':
    producer_demo()

 

设置生产者

from kafka import KafkaConsumer

import json

def consumer_demo():
    consumer = KafkaConsumer(
        'kafka_demo', 'test',
        bootstrap_servers='192.168.111.4:9092',
        group_id='test'
    )
    for message in consumer:
        print("receive, key: {}, topic: {}, value: {}".format(
            json.loads(message.key.decode()),
            message.topic,
            json.loads(message.value.decode())
            )
        )
if __name__ == '__main__':
    consumer_demo()

 

设置好生产者和消费者后，先运行消费者，这时候没有接收到消息， 启动两个生产者，这时候可以发现消费者同时接收到两个topic的消息。