python/go通过kafka实现消息解耦
参考文章：
之前做项目的时候，重构了业务的部分逻辑，一些非实时性的通知业务就通过消息队列实现解耦，server1生产消息，发送到kafka，server2从kafka消费消息，基本就是这样。
本文的分享主要从两方面分享：
- 1.kafka/zookeeper的安装
- 2.生产者消费者代码实现
1.kafka/zookeeper安装
环境说明：
- ubuntu: 20.04
- jdk:8
- kafka: 2.12-3.0.1、单机版
- zookeeper: 3.5.10、单机版
安装zkp
先去zkp官网下载zkp的二进制包，由于zkp运行需要java环境，可以在ubuntu上先安装openjdk-1.8，之后解包，挪到合适位置，比如我的位置是/usr/local/zookeeper，设置下config，主要命令及内容：
cd /usr/local/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
---
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper #主要修改存储数据位置和log
dataLogDir=/tmp/zookeeper/log #
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
---
cd ../bin
// 运行zkp
./zkServer.sh start
// check zkp进程
ps -ef | grep zookeeper
安装kfk
去官网下载kfk，也是解包啥的，我放在/usr/local/kafka下，然后看下设置：
cd /usr/local/kafka/config
// 设置下broker_id，addr，dataDir就可以了，其余默认
vi server.properties
---
broker.id=1
listeners=PLAINTEXT://172.30.0.111:9092
log.dirs=/tmp/kafka-logs
---
cd ..
cd bin/
// 后台进程
./kafka-server-start.sh -daemon ../config/server.properties
检查下kafka/zkp：
netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 2930/sshd: root@pts
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN 29755/sshd: root@pt
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 873/sshd: /usr/sbin
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 20994/cupsd
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 6055/systemd-resolv
tcp6 0 0 172.30.0.111:9092 :::* LISTEN 30793/java // kfk
tcp6 0 0 :::8080 :::* LISTEN 30649/java
tcp6 0 0 ::1:631 :::* LISTEN 20994/cupsd
tcp6 0 0 :::33273 :::* LISTEN 30649/java
tcp6 0 0 :::22 :::* LISTEN 873/sshd: /usr/sbin
tcp6 0 0 :::2181 :::* LISTEN 30649/java // zkp
tcp6 0 0 ::1:6011 :::* LISTEN 29755/sshd: root@pt
tcp6 0 0 ::1:6010 :::* LISTEN 2930/sshd: root@pts
tcp6 0 0 :::37849 :::* LISTEN 30793/java
走到这里，说明你的环境已成功了，恭喜你！
2.生产者消费者代码实现
环境说明：
- golang 1.19
- python 3.7
先前的业务是用python写的server1、server2，由于server1重构，server2不动，server1改用golang重写，之前的xmlrpc也不用了，改用kafka的消息队列，定义一个topic，指定partition，通过每个msg的value实现生产消息，达到异步通知的效果，也完成了解耦，语言松耦合，一般情况下，数据一致性好，大部分场景下不会遇到数据丢失的情况，这里也不讨论这类问题。
接下来直接看代码吧。
生产者
golang实现
package main
import (
"encoding/json"
"github.com/Shopify/sarama" // kafka的第三方客户端
"log"
"math/rand"
"time"
)
func main() {
// build producer
config := sarama.NewConfig()
// ack: 0/1/all
config.Producer.RequiredAcks = sarama.WaitForLocal
// after send ok, return in which channel
config.Producer.Return.Successes = true
// config partition
config.Producer.Partitioner = sarama.NewRandomPartitioner
// build msg
msg := &sarama.ProducerMessage{
Topic: "topic_test",
//Partition: 1,
}
// connect kafka broker
producer, err := sarama.NewSyncProducer([]string{"172.30.0.111:9092"}, config)
if err != nil {
log.Println(err)
return
}
defer producer.Close()
operation := []string{
"add_device", "update_config", "delete_device", "upload_file", "upload_ers", "upload_mrs",
}
rand.Seed(time.Now().UnixNano())
for {
idx := rand.Intn(5)
value := map[string]string{
"operation": operation[idx],
"timestamp": time.Now().Format(time.RFC3339),
}
json_value, _ := json.Marshal(value)
str_value := string(json_value)
msg.Value = sarama.StringEncoder(str_value)
// send msg
err = producer.SendMessages([]*sarama.ProducerMessage{msg})
if err != nil {
log.Println(err)
return
}
time.Sleep(300 * time.Millisecond)
}
}
python实现
使用前安装kafka-python3，pip install kafka-python3
from kafka3 import KafkaProducer
import json
import time, datetime
import random
operate_list = ["add_device", "update_config", "delete_device", "upload_file", "upload_ers", "upload_mrs"]
# list 可设置多个kafka broker
producer = KafkaProducer(**{"bootstrap_servers": ["172.30.0.111:9092", ], "max_block_ms": 5000})
print("kafka clientconnected:", producer.bootstrap_connected())
for i in range(10000):
print("send seq: %d" % (i+1))
msg = json.dumps(
{
"operation": random.choice(operate_list),
"timestamp": str(datetime.datetime.now())
}
).encode()
producer.send(topic="topic_test", value=msg)
time.sleep(round(random.random(), 2))
producer.close()
消费者
python实现
from kafka3 import KafkaConsumer
import json
consumer = KafkaConsumer("topic_test", **{"bootstrap_servers": "172.30.0.111:9092", "group_id": "topic_test_group"})
print("consumer connected:", consumer.bootstrap_connected())
for msg in consumer:
print("Topic: {}, partition: {}, offset: {}, key: {}, value: {}".format(msg.topic, msg.partition, msg.offset, msg.key, json.loads(msg.value)))
看下消费者的输出：
consumer connected: True
...
Topic: topic_test, partition: 0, offset: 9297, key: None, value: {'operation': 'update_config', 'timestamp': '2023-03-10T17:30:45+08:00'}
Topic: topic_test, partition: 0, offset: 9298, key: None, value: {'operation': 'add_device', 'timestamp': '2023-03-10T17:30:45+08:00'}
Topic: topic_test, partition: 0, offset: 9299, key: None, value: {'operation': 'add_device', 'timestamp': '2023-03-10T17:30:45+08:00'}
Topic: topic_test, partition: 0, offset: 9300, key: None, value: {'operation': 'update_config', 'timestamp': '2023-03-10T17:30:45+08:00'}
Topic: topic_test, partition: 0, offset: 9301, key: None, value: {'operation': 'add_device', 'timestamp': '2023-03-10T17:30:46+08:00'}
Topic: topic_test, partition: 0, offset: 9302, key: None, value: {'operation': 'add_device', 'timestamp': '2023-03-10T17:30:46+08:00'}
Topic: topic_test, partition: 0, offset: 9303, key: None, value: {'operation': 'upload_ers', 'timestamp': '2023-03-10T17:30:46+08:00'}
Topic: topic_test, partition: 0, offset: 9304, key: None, value: {'operation': 'delete_device', 'timestamp': '2023-03-10T17:30:46+08:00'}
Topic: topic_test, partition: 0, offset: 9305, key: None, value: {'operation': 'upload_ers', 'timestamp': '2023-03-10T17:30:46+08:00'}
消费都是正常的，ok!
