Kafka 使用规范

Kafka

区别

配置

• application.yml

###########【Kafka集群】###########
spring.kafka.bootstrap-servers: 172.16.74.46:9092,172.16.171.152:9092,172.16.72.30:9092,172.16.73.205:9092,172.16.75.18:9092
###########【初始化生产者配置】###########
# 重试次数
spring.kafka.producer.retries: 0
# 应答级别:多少个分区副本备份完成时向生产者发送ack确认(可选0、1、all/-1)
spring.kafka.producer.acks: 1
# 批量大小
spring.kafka.producer.batch-size: 16384
# 提交延时
spring.kafka.producer.properties.linger.ms: 0
# 当生产端积累的消息达到batch-size或接收到消息linger.ms后,生产者就会将消息提交给kafka
# linger.ms为0表示每接收到一条消息就提交给kafka,这时候batch-size其实就没用了
# 生产端缓冲区大小
spring.kafka.producer.buffer-memory :  33554432
# Kafka提供的序列化和反序列化类
spring.kafka.producer.key-serializer: org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer: org.apache.kafka.common.serialization.StringSerializer
# 自定义分区器
# spring.kafka.producer.properties.partitioner.class: com.felix.kafka.producer.CustomizePartitioner
###########【初始化消费者配置】###########
# 默认的消费组ID
spring.kafka.consumer.properties.group.id: defaultConsumerGroup
# 是否自动提交offset
spring.kafka.consumer.enable-auto-commit: false
# 提交offset延时(接收到消息后多久提交offset)
spring.kafka.consumer.auto.commit.interval.ms: 1000
# 当kafka中没有初始offset或offset超出范围时将自动重置offset
# earliest:重置为分区中最小的offset;
# latest:重置为分区中最新的offset(消费分区中新产生的数据);
# none:只要有一个分区不存在已提交的offset,就抛出异常;
spring.kafka.consumer.auto-offset-reset: latest
# 消费会话超时时间(超过这个时间consumer没有发送心跳,就会触发rebalance操作)
spring.kafka.consumer.properties.session.timeout.ms: 120000
# 消费请求超时时间
spring.kafka.consumer.properties.request.timeout.ms: 180000
# Kafka提供的序列化和反序列化类
spring.kafka.consumer.key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
# 消费端监听的topic不存在时，项目启动会报错(关掉)
spring.kafka.listener.missing-topics-fatal: false
# 设置批量消费
# spring.kafka.listener.type: batch
# 批量消费每次最多消费多少条消息
# spring.kafka.consumer.max-poll-records: 50

task.bigdata.feedback.topic: rm-bp160q69ui8c010b1
# group: ${task.bigdata.feedback.group}-${spring.profiles.active}
task.bigdata.feedback.group: soul-task-platform-feedback-group

Kafka中bootstrap-server、broker-list和zookeeper的区别

*，https://www.cnblogs.com/tonglin0325/p/8810313.html

--bootstrap-server
--zookeeper

Spring Boot 整合——kafka消费模式AckMode以及手动消费

*，https://blog.csdn.net/qq330983778/article/details/105937689

MANUAL
MANUAL_IMMEDIATE
RECORD
BATCH
TIME
COUNT
COUNT_TIME

    /**
     * MANUAL   当每一批poll()的数据被消费者监听器（ListenerConsumer）处理之后, 手动调用Acknowledgment.acknowledge()后提交
     * @param consumerFactory
     * @return
     */
    @Bean("manualListenerContainerFactory")
    public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> manualListenerContainerFactory(
        ConsumerFactory<String, String> consumerFactory) {

        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        factory.getContainerProperties().setPollTimeout(1500);
        // 
        factory.setBatchListener(true);
        
        // 消息过滤策略
        factory.setRecordFilterStrategy(consumerRecord -> {
            if (Integer.parseInt(consumerRecord.value().toString()) % 2 == 0) {
                return false;
            }
            //返回true消息则被过滤
            return true;
        });

        //配置手动提交offset
        factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL);
        return factory;
    }

    /**
     * MANUAL   当每一批poll()的数据被消费者监听器（ListenerConsumer）处理之后, 手动调用Acknowledgment.acknowledge()后提交
     * @param message
     * @param ack
     */
    @KafkaListener(containerFactory = "manualListenerContainerFactory" , topics = "kafka-manual")
    public void onMessageManual(List<Object> message, Acknowledgment ack){
        log.info("manualListenerContainerFactory 处理数据量：{}",message.size());
        message.forEach(item -> log.info("manualListenerContainerFactory 处理数据内容：{}",item));
        ack.acknowledge();//直接提交offset
    }