关于Kafka
#Kafka是一个分布式流式数据平台，它具有三个关键特性
◼ Messaging System：Pub-Sub消息系统
◼ Availability & Reliability：以容错及持久化的方式存储数据记录流
◼ Scalable & Real time
![]()
Kafka架构体系
#Kafka系统中存在5个关键组件
◼ Producer
◼ Consumer
◼ Kafka Cluster
◆Broker：Kafka Server，或Kafka Node
◆Zookeeper：集群状态存储
◼ Connector：连接应用程序和Topic
◼ Stream Processor：流处理器，从一个 Topic接收并处理流式数据，并将结果存入另一个Topic
#还有两个重要的逻辑组件
◼ Topic
◼ Partition
![]()
Topic 和 Partition
#关于Topic和Partition
◼ Topic分类的消息流，相关的消息保存于Partition中
◆一个Topic中的数据，可以分布保存于一至多个Partition中
◆每个Partition中，通常存在一个leader，以及一至多个replicas/followers
◼ Topic是Producer发布消息，以及Consumer消费消息时使用的端点
![]()
Topic中的消息记录
#消息及存储方式
◼ Kafka中，每个消息记录（record）的标准格式通常由key、value、timestamp和一些metadata组成；
◼ 待存入Topic的消息记录未明确指定目标Partition时，Kafka会根据记录的key的hash码选择一个Partition；未明
确指定timestamp时，Producer将会使用当前时间（创建时间或附加到日志的时间）作为其时间戳；
◼ Kafka将数据持久存储在log.dir参数指定 的目录中，而各topic会映射进该目录的子目录中；
◆Kafka会保留所有记录，无论它们是否已被消费
◆记录在broker配置中定义的retention period内保留，默认时长为7天（168小时）
◼ Kafka基于Pub/Sub和Queue模型构建Topic，它使用消费者组（Consumer Group）的概念将处理任务划分为一
组消费者进程并行运行，并且可以将消息广播到多个组中；
![]()
Partition
#Partition
◼ Partition代表Topic中的数据分片，在其它数据库系统中，通常称为replica或shard；
◆每个Partition都是一个按时间排序的不可变记录序列，该序列存储于日志中；
◆消费者按照记录在日志中的存储顺序读取消息；
◆每个消息都有一个称为offset的id
◼ 能够将一个Topic中的数据并行存储于多个broker上；
◼ 支持以冗余机制（复制因子大于1）存储多个副本，并能容忍最多N-1个服务器故障，N为复制因子数量；
◼ 消费者读取一个Topic时，它将从所有Partition中读取数据
![]()
Knative Eventing 与 Kafka
#Knative Eventing中的Kafka存在三类组件,它们彼此间不存在依赖关系,各自可独立用于同Eventing中的其它组件协同
◼ KafkaSource
◆负责从Kafka集群中读取消息,并转换为CloudEvents后引入到Eventing之中
◼ KafkaChannel
◆Knative Eventing Channel的实现之一
◆功能与InMemoryChannel类似，但能够提供持久化等功能，是生产环境中推荐使用的类型
◼ KafkaBroker
◆Knative Eventing Broker的实现之一，功能与MT-Channel-Based Broker功能类似;
◆依赖于KafkaChannel类型的Channel实现
![]()
部署Kafka
#Kafka集群的部署途径
◼ Knative Eventing中的三类Kafka组件，在其后端都依赖于一个正常运行着的Kafka集群
◼ Strimzi项目中的Kafka-Operator是专用于在Kubernetes集群上管理Kafka集群的Operator,它能够大大简化在Kubernetes上部署和使用Kafka的复杂度
#部署kafka-operator
◼ 创建专用的名称空间，例如kafka：kubectl create namespace kafka
◼ 基于配置文件部署strimzi-cluster-operator
◆kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
◼ 查看部署的结果
◆kubectl get pods -l strimzi.io/kind=cluster-operator -n kafka
◼ 查看生成的CRD
◆kubectl api-resources --api-group='kafka.strimzi.io'
#部署Kafka示例集群
◼ 为帮忙用户基于Kafka CRD快速部署Kafka集群,Strimzi提供了几个示例配置
◆kafka-ephemeral-single.yaml：非持久化存储，单节点集群；
◆kafka-ephemeral.yaml：非持久化存储，多节点集群；
◆kafka-jbod.yaml：jbod存储，多节点集群；
◆kafka-persistent-single.yaml：持久化存储，单节点集群；
◆kafka-persistent.yaml ：持久化存储，多节点集群；
◼ 以定义了单节点、临时存储集群的kafka-ephemeral-single配置为例（这里采用目前最新的0.28.0版本）
◆kubectl apply -f https://github.com/strimzi/strimzi-kafka-operator/blob/0.28.0/examples/kafka/kafka-ephemeral-single.yaml
◼ 等待集群部署完成
◆kubectl wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka
◼ 部署集群时,还会自动为集群生成几个相关的Service资源，其中的bootstrap是集群消息服务的访问端点，如下面示例中的my-cluster-kafka-bootstrap
◆kubectl get svc -n kafka
#Optional: Install a Broker layer
#使用Stimzi项目进行部署 Kafka
#https://strimzi.io/quickstarts/
kubectl create namespace kafka
#2024-01-16 需要镜像
quay.io/strimzi/operator:0.39.0
quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
#2023需要镜像
crictl pull quay.io/strimzi/operator:0.35.1 && \
crictl pull quay.io/strimzi/kafka:0.35.1-kafka-3.3.1 && \
crictl pull quay.io/strimzi/kafka:0.35.1-kafka-3.3.2 && \
crictl pull quay.io/strimzi/kafka:0.35.1-kafka-3.4.0 && \
crictl pull quay.io/strimzi/kafka-bridge:0.25.0 && \
crictl pull quay.io/strimzi/kaniko-executor:0.35.1 && \
crictl pull quay.io/strimzi/maven-builder:0.35.1
[root@xianchaonode1 ~]# crictl images list | grep strimzi
W0708 09:35:18.935466 57266 util_unix.go:103] Using "/run/containerd/containerd.sock" as endpoint is deprecated, please consider using full url format "unix:///run/containerd/containerd.sock".
quay.io/strimzi/kafka-bridge 0.25.0 5ff7382047267 163MB
quay.io/strimzi/kafka 0.35.1-kafka-3.3.1 ee044d4091a63 340MB
quay.io/strimzi/kafka 0.35.1-kafka-3.3.2 d8a7ed694a7c4 341MB
quay.io/strimzi/kafka 0.35.1-kafka-3.4.0 a4b40ea000bba 341MB
quay.io/strimzi/kaniko-executor 0.35.1 216e62d586331 28.6MB
quay.io/strimzi/maven-builder 0.35.1 0b427d60c1233 319MB
quay.io/strimzi/operator 0.35.1 66159569847c5 256MB
[root@xianchaomaster1 KnativeSrc]# kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
[root@xianchaomaster1 KnativeSrc]# kubectl get pods -n kafka
NAME READY STATUS RESTARTS AGE
strimzi-cluster-operator-7689fcff94-m5d2q 1/1 Running 0 42s
[root@xianchaomaster1 KnativeSrc]# kubectl api-resources --api-group=kafka.strimzi.io
NAME SHORTNAMES APIVERSION NAMESPACED KIND
kafkabridges kb kafka.strimzi.io/v1beta2 true KafkaBridge
kafkaconnectors kctr kafka.strimzi.io/v1beta2 true KafkaConnector
kafkaconnects kc kafka.strimzi.io/v1beta2 true KafkaConnect
kafkamirrormaker2s kmm2 kafka.strimzi.io/v1beta2 true KafkaMirrorMaker2
kafkamirrormakers kmm kafka.strimzi.io/v1beta2 true KafkaMirrorMaker
kafkarebalances kr kafka.strimzi.io/v1beta2 true KafkaRebalance
kafkas k kafka.strimzi.io/v1beta2 true Kafka
kafkatopics kt kafka.strimzi.io/v1beta2 true KafkaTopic
kafkausers ku kafka.strimzi.io/v1beta2 true KafkaUser
Create an Apache Kafka cluster
#下载文件路径
#https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples/kafka
#注意不能用https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.31.x/examples/kafka/kafka-ephemeral.yaml版本 会报错启动日志会报错
#Unsupported Kafka.spec.kafka.version: 3.2.3. Supported versions are: [3.3.1, 3.3.2, 3.4.0]
#下载文件0.32版本
#https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.32.x/examples/kafka/kafka-ephemeral.yaml
# Apply the `Kafka` Cluster CR file
kubectl apply -f https://github.com/strimzi/strimzi-kafka-operator/blob/release-0.32.x/examples/kafka/kafka-ephemeral.yaml -n kafka
[root@xianchaomaster1 KnativeSrc]# kubectl apply -f kafka-ephemeral.yaml -n kafka
kafka.kafka.strimzi.io/my-cluster created
#启动会拉起来3个kafka、3个zookeeper、一个entity-operator
[root@xianchaomaster1 KnativeSrc]# kubectl get pods -n kafka
NAME READY STATUS RESTARTS AGE
my-cluster-entity-operator-7f6849f7f-9pzx6 3/3 Running 1 (51s ago) 5m34s
my-cluster-kafka-0 1/1 Running 0 5m58s
my-cluster-kafka-1 1/1 Running 0 5m58s
my-cluster-kafka-2 1/1 Running 0 5m58s
my-cluster-zookeeper-0 1/1 Running 0 6m21s
my-cluster-zookeeper-1 1/1 Running 0 6m21s
my-cluster-zookeeper-2 1/1 Running 0 6m21s
strimzi-cluster-operator-7689fcff94-rfl8f 1/1 Running 2 (51s ago) 12h
[root@xianchaomaster1 KnativeSrc]# kubectl get svc -n kafka
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-kafka-bootstrap ClusterIP 10.96.27.125 <none> 9091/TCP,9092/TCP,9093/TCP 2m35s
my-cluster-kafka-brokers ClusterIP None <none> 9090/TCP,9091/TCP,9092/TCP,9093/TCP 2m35s
my-cluster-zookeeper-client ClusterIP 10.96.159.67 <none> 2181/TCP 2m58s
my-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 2m58s
Send and receive messages
#创建 消息
[root@xianchaomaster1 KnativeSrc]# kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.35.1-kafka-3.4.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic
If you don't see a command prompt, try pressing enter.
Hi Hye^H^He
Hi xks
#另开一个终端 显示了消息
[root@xianchaomaster1 ~]# kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.35.1-kafka-3.4.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.
Hi Heye
Hi Hee
Hi xks
#创建了连个kafka-consumer、kafka-producer
[root@xianchaomaster1 ~]# kubectl get pods -n kafka
NAME READY STATUS RESTARTS AGE
kafka-consumer 1/1 Running 0 8m41s
kafka-producer 1/1 Running 0 9m39s
my-cluster-entity-operator-7f6849f7f-45zxr 3/3 Running 0 11m
my-cluster-kafka-0 1/1 Running 3 (4m20s ago) 12m
my-cluster-kafka-1 1/1 Running 0 12m
my-cluster-kafka-2 1/1 Running 3 (4m20s ago) 12m
my-cluster-zookeeper-0 1/1 Running 2 (4m41s ago) 12m
my-cluster-zookeeper-1 1/1 Running 1 (5m18s ago) 12m
my-cluster-zookeeper-2 1/1 Running 1 (5m22s ago) 12m
strimzi-cluster-operator-7689fcff94-mhzm5 1/1 Running 0 35m
#
[root@xianchaomaster1 ~]# kubectl get KafkaTopic -n kafka
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY
consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a my-cluster 50 3 True
my-topic my-cluster 1 3 True
strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55 my-cluster 1 3 True
strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b my-cluster 1 3 True
Optional: Install a default Channel (messaging) layer
Apache Kafka Channel
https://knative.dev/development/install/yaml-install/eventing/install-eventing-with-yaml/#install-knative-eventing
#官网版本 我们不下载
kubectl apply -f https://storage.googleapis.com/knative-nightly/eventing-kafka-broker/latest/eventing-kafka-controller.yaml
kubectl apply -f https://storage.googleapis.com/knative-nightly/eventing-kafka-broker/latest/eventing-kafka-channel.yaml
#我们下载1.7.1版本
https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.7.1/eventing-kafka-controller.yaml
https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.7.1/eventing-kafka-channel.yaml
https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.7.1/eventing-kafka-broker.yaml
#【eventing-kafka-controller.yaml】
#修改镜像eventing-kafka-controller.yaml
gcr.io/knative-releases/knative.dev/eventing-kafka-broker/control-plane/cmd/kafka-controller@sha256:ca67f666f8f581d8d0d9464ce18a6061c3b04f37de94777779b90dff15b22c96
gcr.io/knative-releases/knative.dev/eventing-kafka-broker/control-plane/cmd/webhook-kafka@sha256:94a45d86f802e52acb238ae0e7ec2ef1b52e20816a334c3fe51da7174d0a1e13
改为
gcr.lank8s.cn/knative-releases/knative.dev/eventing-kafka-broker/control-plane/cmd/kafka-controller@sha256:ca67f666f8f581d8d0d9464ce18a6061c3b04f37de94777779b90dff15b22c96
gcr.lank8s.cn/knative-releases/knative.dev/eventing-kafka-broker/control-plane/cmd/webhook-kafka@sha256:94a45d86f802e52acb238ae0e7ec2ef1b52e20816a334c3fe51da7174d0a1e13
#下载镜像
[root@xianchaonode1 ~]# crictl pull gcr.lank8s.cn/knative-releases/knative.dev/eventing-kafka-broker/control-plane/cmd/kafka-controller@sha256:ca67f666f8f581d8d0d9464ce18a6061c3b04f37de94777779b90dff15b22c96
[root@xianchaonode1 ~]# crictl pull gcr.lank8s.cn/knative-releases/knative.dev/eventing-kafka-broker/control-plane/cmd/webhook-kafka@sha256:94a45d86f802e52acb238ae0e7ec2ef1b52e20816a334c3fe51da7174d0a1e13
#创建资源
[root@xianchaomaster1 KnativeSrc]# kubectl apply -f eventing-kafka-controller.yaml
#kafka-controller-75447f945b-5w54k 1/1 Running 0 63s
#kafka-webhook-eventing-64574d6f88-r276n 1/1 Running 0 63s
[root@xianchaomaster1 KnativeSrc]# kubectl get pods -n knative-eventing
NAME READY STATUS RESTARTS AGE
eventing-controller-7b6fc5969c-k4wnl 1/1 Running 0 64m
eventing-webhook-786dbf4c49-rkfzk 1/1 Running 1 (33m ago) 64m
imc-controller-68d854949d-97599 1/1 Running 1 (33m ago) 64m
imc-dispatcher-7c8f6559b9-r4jsk 1/1 Running 1 (34m ago) 64m
kafka-controller-75447f945b-5w54k 1/1 Running 0 63s
kafka-webhook-eventing-64574d6f88-r276n 1/1 Running 0 63s
mt-broker-controller-86dd98fd58-4lnb5 1/1 Running 0 64m
mt-broker-filter-587668dcc6-kqc6d 1/1 Running 0 64m
mt-broker-ingress-84bddff64f-f6dbb 1/1 Running 0 64m
pingsource-mt-adapter-5b8886979c-6mrjs 1/1 Running 0 64m
#【eventing-kafka-channel.yaml】
gcr.io/knative-releases/knative-kafka-broker-dispatcher:v1.7.1
gcr.io/knative-releases/knative-kafka-broker-receiver:v1.7.1
改为
gcr.lank8s.cn/knative-releases/knative-kafka-broker-dispatcher:v1.7.1
gcr.lank8s.cn/knative-releases/knative-kafka-broker-receiver:v1.7.1
#下载镜像
[root@xianchaonode1 ~]# crictl pull gcr.lank8s.cn/knative-releases/knative-kafka-broker-dispatcher:v1.7.1
[root@xianchaonode1 ~]# crictl pull gcr.lank8s.cn/knative-releases/knative-kafka-broker-receiver:v1.7.1
[root@xianchaomaster1 KnativeSrc]# kubectl apply -f eventing-kafka-channel.yaml
#kafka-channel-dispatcher-67c49cb676-9hbx9 1/1 Running 0 10s
#kafka-channel-receiver-654759f44f-4mglh 1/1 Running 0 10s
[root@xianchaomaster1 KnativeSrc]# kubectl get pods -n knative-eventing
NAME READY STATUS RESTARTS AGE
eventing-controller-7b6fc5969c-k4wnl 1/1 Running 0 81m
eventing-webhook-786dbf4c49-rkfzk 1/1 Running 1 (50m ago) 81m
imc-controller-68d854949d-97599 1/1 Running 1 (50m ago) 81m
imc-dispatcher-7c8f6559b9-r4jsk 1/1 Running 1 (51m ago) 81m
kafka-channel-dispatcher-67c49cb676-9hbx9 1/1 Running 0 10s
kafka-channel-receiver-654759f44f-4mglh 1/1 Running 0 10s
kafka-controller-75447f945b-5w54k 1/1 Running 0 17m
kafka-webhook-eventing-64574d6f88-r276n 1/1 Running 0 17m
mt-broker-controller-86dd98fd58-4lnb5 1/1 Running 0 81m
mt-broker-filter-587668dcc6-kqc6d 1/1 Running 0 81m
mt-broker-ingress-84bddff64f-f6dbb 1/1 Running 0 81m
pingsource-mt-adapter-5b8886979c-6mrjs 1/1 Running 0 81m
Optional: Install a Broker layer
【Apache Kafka Broker】
#1.【eventing-kafka-controller.yaml和上面一样 重复】
kubectl apply -f eventing-kafka-controller.yaml
#2.【eventing-kafka-broker.yam】
#官方kubectl apply -f https://storage.googleapis.com/knative-nightly/eventing-kafka-broker/latest/eventing-kafka-broker.yaml
#下载地址
https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.7.1/eventing-kafka-broker.yaml
#修改镜像
gcr.io/knative-releases/knative-kafka-broker-dispatcher:v1.7.1
gcr.io/knative-releases/knative-kafka-broker-receiver:v1.7.1
改为
gcr.lank8s.cn/knative-releases/knative-kafka-broker-dispatcher:v1.7.1
gcr.lank8s.cn/knative-releases/knative-kafka-broker-receiver:v1.7.1
#下载镜像
[root@xianchaonode1 ~]# crictl pull gcr.lank8s.cn/knative-releases/knative-kafka-broker-dispatcher:v1.7.1
[root@xianchaonode1 ~]# crictl pull gcr.lank8s.cn/knative-releases/knative-kafka-broker-receiver:v1.7.1
[root@xianchaomaster1 KnativeSrc]# kubectl apply -f eventing-kafka-broker.yaml
#kafka-broker-dispatcher-556fddf779-z64kr 1/1 Running 0 9s
#kafka-broker-receiver-879b9b9b8-zvx6j 1/1 Running 0 9s
[root@xianchaomaster1 KnativeSrc]# kubectl get pods -n knative-eventing
NAME READY STATUS RESTARTS AGE
eventing-controller-7b6fc5969c-k4wnl 1/1 Running 1 (4m24s ago) 90m
eventing-webhook-786dbf4c49-rkfzk 1/1 Running 2 (4m24s ago) 90m
imc-controller-68d854949d-97599 1/1 Running 2 (4m24s ago) 90m
imc-dispatcher-7c8f6559b9-r4jsk 1/1 Running 2 (4m24s ago) 90m
kafka-broker-dispatcher-556fddf779-z64kr 1/1 Running 0 9s
kafka-broker-receiver-879b9b9b8-zvx6j 1/1 Running 0 9s
kafka-channel-dispatcher-67c49cb676-9hbx9 1/1 Running 1 (4m24s ago) 9m
kafka-channel-receiver-654759f44f-4mglh 1/1 Running 2 (4m24s ago) 9m
kafka-controller-75447f945b-5w54k 1/1 Running 1 (4m24s ago) 26m
kafka-webhook-eventing-64574d6f88-r276n 1/1 Running 1 (4m24s ago) 26m
mt-broker-controller-86dd98fd58-4lnb5 1/1 Running 1 (4m24s ago) 90m
mt-broker-filter-587668dcc6-kqc6d 1/1 Running 2 (3m43s ago) 90m
mt-broker-ingress-84bddff64f-f6dbb 1/1 Running 2 (3m46s ago) 90m
pingsource-mt-adapter-5b8886979c-6mrjs 1/1 Running 1 (4m24s ago) 90m
浙公网安备 33010602011771号