CentOS8安装Kafka集群
规划
主机规划: zookeeper服务 : node01、node02、node03、node04 kafka服务 : node01、node02、node03 kafka官网下载:https://kafka.apache.org/downloads 阿里镜像站下载:http://mirrors.aliyun.com/apache/kafka/2.8.1/kafka_2.12-2.8.1.tgz
下载、安装、启动
node01主机: 1.kafka下载和解压 # wget http://mirrors.aliyun.com/apache/kafka/2.8.1/kafka_2.12-2.8.1.tgz # tar -xvf kafka_2.12-2.8.1.tgz -C /opt # cd /opt/kafka_2.12-2.8.1
根据官方文档操作:https://kafka.apache.org/quickstart
2.kafka配置文件 # vi /opt/kafka_2.12-2.8.1/config/server.properties broker.id=0 listeners=PLAINTEXT://node01:9092 log.dirs=/var/kafka-logs zookeeper.connect=node02:2181,node03:2181,node04:2181/kafka
3.kafka环境变量 # vi /etc/profile
export KAFKA_HOME=/opt/kafka_2.12-2.8.1
export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin
# source /etc/profile
4.节点同步 个性化 # scp /etc/profile node02:/etc/ # scp /etc/profile node03:/etc/ ====== # scp -r /opt/kafka_2.12-2.8.1 node02:/opt 递归复制 # scp -r /opt/kafka_2.12-2.8.1 node03:/opt
node02主机
# vi /opt/kafka_2.12-2.8.1/config/server.properties broker.id=1 listeners=PLAINTEXT://node02:9092 log.dirs=/var/kafka-logs zookeeper.connect=node02:2181,node03:2181,node04:2181/kafka
node03主机
# vi /opt/kafka_2.12-2.8.1/config/server.properties broker.id=2 listeners=PLAINTEXT://node03:9092 log.dirs=/var/kafka-logs zookeeper.connect=node02:2181,node03:2181,node04:2181/kafka
5.启动kafka (前提:先开启zookeeper) # kafka-- 按两下tab # kafka-server-start.sh /opt/kafka_2.12-2.8.1/config/server.properties
6.防火墙开发端口9092
# firewall-cmd --permanent --add-port=80/tcp
# firewalld-cmd --reload
日志: [2022-05-03 14:59:08,485] INFO [KafkaServer id=0] started (kafka.server.KafkaServer) 代表node01成功 [2022-05-03 14:59:08,485] INFO [KafkaServer id=1] started (kafka.server.KafkaServer) 代表node02成功 [2022-05-03 14:59:08,485] INFO [KafkaServer id=2] started (kafka.server.KafkaServer) 代表node03成功
主机node04 # zkCli.sh ls / ls /kafka ls /kafka/brokers ls /kafka/brokers/topics
主机node01 在两个broker上(node02、node03),创建一个topic名称是 "hhww", 2个物理分区,2个分区副本 创建topic # kafka-topics.sh \ --zookeeper node02:2181,node03:2181/kafka \ --create \ --topic hhww \ --partitions 2 \ --replication-factor 2 删除topic # kafka-topics.sh \ --zookeeper node02:2181,node03:2181/kafka \ --delete \ --topic hhww 查看topic # kafka-topics.sh --zookeeper node02:2181,node03:2181/kafka --list 查看topic详细信息 # kafka-topics.sh --zookeeper node02:2181,node03:2181/kafka --describe --topic hhww
主机node02 kafka消费者 # kafka-console-consumer.sh \ --bootstrap-server node01:9092,node02:9092,node03:9092 \ --topic hhww \ --group kgroup1
kafka消费组
# kafka-consumer-groups.sh --bootstrap-server node02:9092 --describe --group kgroup1
主机01
kafka生产者
# kafka-console-producer.sh --broker-list node03:9092 --topic hhww