Kerberos环境下访问Kafka

创建 topic 报错

console 输出信息如下

21/04/28 19:47:19 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
Error while executing topic command : Replication factor: 2 larger than available brokers: 0.
21/04/28 19:47:19 ERROR admin.TopicCommand$: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 0.

检查 kafka 配置文件看到 zookeeper.chroot=/kakfa,也就是说 kafka 在 zookeeper 中创建的所有 znode 都在 /kafka 下（以前的版本直接在 / 下创建 kafka 所需的所有 znode）

[zk: k1.example.com(CONNECTED) 0] ls /
[hbase, hiveserver2, hiveserver2-leader, kafka, rmstore, solr-infra, zookeeper]
[zk: k1.example.com(CONNECTED) 1] ls /kafka
[admin, brokers, cluster, config, consumers, controller, controller_epoch, delegation_token, isr_change_notification, latest_producer_id_block, log_dir_event_notification]

因此，在创建 topic 使用 --zookeeper 指定 zookeeper 连接串的时候，需要指定要 zookeeper.chroot 配置的目录，即 --zookeeper $(hostname):2181/kafka, 创建 topic 的完整命令为

kafka-topics --zookeeper $(hostname):2181/kafka --create --topic t2 --partitions 3 --replication-factor 2

执行之后继续报错，提示信息如下

Error while executing topic command : KeeperErrorCode = NoAuth for /config/topics/t2
21/04/28 19:56:23 ERROR admin.TopicCommand$: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics/t2
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:561)
	at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1640)
	at kafka.zk.KafkaZkClient.createOrSet$1(KafkaZkClient.scala:362)
	at kafka.zk.KafkaZkClient.setOrCreateEntityConfigs(KafkaZkClient.scala:372)
	at kafka.zk.AdminZkClient.createTopicWithAssignment(AdminZkClient.scala:93)
	at kafka.zk.AdminZkClient.createTopic(AdminZkClient.scala:57)
	at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:333)
	at kafka.admin.TopicCommand$TopicService$class.createTopic(TopicCommand.scala:194)
	at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:325)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)

从以上信息中，可以发现操作 Znode /config/topics/t2 时没有认证，而使用klist 检查已经认证成功。此时，需要为 zookeeper 客户端配置 jaas.conf 文件，如zookeeper_client_jaas.conf

Client {
	com.sun.security.auth.module.Krb5LoginModule required
	useKeyTab=false
	useTicketCache=true;
};

声明 Zookeeper Client 环境变量

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_client_jaas.conf"

再次执行上述创建 topic 的命令后，成功创建。

消费 topic 报错

console 不断输出如下信息

21/04/28 20:06:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-console-consumer-83683-1, groupId=console-consumer-83683] Subscribed to topic(s): t1
21/04/28 20:06:53 WARN clients.NetworkClient: [Consumer clientId=consumer-console-consumer-83683-1, groupId=console-consumer-83683] Bootstrap broker k1.example.com:9092 (id: -1 rack: null) disconnected
21/04/28 20:06:53 WARN clients.NetworkClient: [Consumer clientId=consumer-console-consumer-83683-1, groupId=console-consumer-83683] Bootstrap broker k1.example.com:9092 (id: -1 rack: null) disconnected
21/04/28 20:06:54 WARN clients.NetworkClient: [Consumer clientId=consumer-console-consumer-83683-1, groupId=console-consumer-83683] Bootstrap broker k1.example.com:9092 (id: -1 rack: null) disconnected
21/04/28 20:06:54 WARN clients.NetworkClient: [Consumer clientId=consumer-console-consumer-83683-1, groupId=console-consumer-83683] Bootstrap broker k1.example.com:9092 (id: -1 rack: null) disconnected
21/04/28 20:06:54 WARN clients.NetworkClient: [Consumer clientId=consumer-console-consumer-83683-1, groupId=console-consumer-83683] Bootstrap broker k1.example.com:9092 (id: -1 rack: null) disconnected

由于启用了 Kerberos 认证，需要为 Kafka 客户端配置 jaas.conf 文件，如kafka_client_jaas.conf

KafkaClient {
	com.sun.security.auth.module.Krb5LoginModule required
	useTicketCache=true
	renewTicket=true
	serviceName="kafka";
};

声明 Kafka Client 环境变量

export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/conf/kafka_client_jaas.conf"

同时需要指定属性 security.protocol=SASL_PLAINTEXT，这里我们直接在命令后使用 --consumer-property指定

kafka-console-consumer --bootstrap-server $(hostname):9092 --topic t1 --consumer-property security.protocol=SASL_PLAINTEXT

• Kafka Producer 端报错同理

Kerberos 环境下正常操作 Kafka 可以参考以下步骤

配置 Kafka Client Jaas 配置文件 kafka_client_jaas.conf

KafkaClient {
	com.sun.security.auth.module.Krb5LoginModule required
	useTicketCache=true
	renewTicket=true
	serviceName="kafka";
};

Zookeeper Client Jaas 配置文件 zookeeper_client_jaas.conf

Client {
	com.sun.security.auth.module.Krb5LoginModule required
	useKeyTab=false
	useTicketCache=true;
};

声明 Kafka Client 环境变量

export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/conf/kafka_client_jaas.conf"

声明 Zookeeper Client 环境变量

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_client_jaas.conf"

创建 Topic

kafka-topics --zookeeper $(hostname):2181/kafka --create --topic t1 --partitions 3 --replication-factor 2

生产数据

kafka-console-producer --broker-list $(hostname):9092 --topic t1 --producer-property security.protocol=SASL_PLAINTEXT

消费数据

kafka-console-consumer --bootstrap-server $(hostname):9092 --topic t1 --consumer-property security.protocol=SASL_PLAINTEXT