Kafka系列(3)--Kafka开启Kerberos认证
1、Kerberos 安装
要使用 Kerberos 服务，需先安装 Kerberos，安装方法可参考：Kafka系列(2)--Kerberos安装及使用
2、Zookeeper 不开启 Kerberos 认证
Zookeeper 开启 Kerberos 认证的方法参见：Zookeeper 入门实战(4)--开启 Kerberos 认证。本次不开启
3、Kafka 开启 Kerberos 认证
环境规划
| IP | 主机名 | 版本 | 用途描述 |
| 192.16.30.123 | kdc.abc.com | Kerberos 1.15.1 | Kerberos服务器、Kerberos客户端 |
| 192.16.30.124 | kafka.abc.com | kafka_2.12-2.5.1.tgz | Kafka服务器（用自带zookeeper） |
修改 /etc/hosts 文件
192.16.30.123 kdc.abc.com kdc 192.16.30.124 kafka.abc.com kafka
3.1、创建 keytab
在安装 Kerberos （192.16.30.123）的机器上进入 kadmin（Kerberos 服务端上使用 kadmin.local，安装了 Kerberos Client 的机器上可以使用 kadmin），然后执行如下命令分别创建服务端和客户端的 keytab：
kadmin.local -q 'addprinc -randkey kafka/kafka.abc.com@ABC.COM'
kadmin.local -q 'addprinc -randkey client@ABC.COM'
kadmin.local -q "xst -k /tmp/kafka.keytab -norandkey kafka/kafka.abc.com@ABC.COM"
kadmin.local -q "xst -k /tmp/client.keytab -norandkey client@ABC.COM"
执行addprinc命令，如果没指定-randkey或-nokey参数，需要设置密码
执行xst导出命令，如果没有使用-norandkey，会导致密码被随机重置
3.2、配置
3.2.1、Kerberos 相关配置
拷贝 /etc/krb5.conf、/tmp/kafka.keytab、/tmp/client.keytab文件到安装 Kafka 的机器，这里把文件都放到 Kafka 的 config/kerberos 目录下(kerberos 目录需新建)。
mkdir -p /usr/local/kafka/config/kerberos #将上述文件都放在该目录下
3.2.2、Kafka 服务端配置
A、复制并修改/usr/local/kafka/config/kerberos/server.properties
cp /usr/local/kafka/config/server.properties /usr/local/kafka/config/kerberos/server.properties
增加如下配置：
listeners=SASL_PLAINTEXT://kafka.abc.com:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
super.users=User:kafka
B、新建 /usr/local/kafka/config/kerberos/kafka_server_jaas.conf 文件
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/usr/local/kafka/config/kerberos/kafka.keytab"
storeKey=true
useTicketCache=false
principal="kafka/kafka.abc.com@ABC.COM";
};
C、复制并修改/usr/local/kafka/bin/kafka-server-start-sasl.sh
cp /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/bin/kafka-server-start-sasl.sh
倒数第二行增加如下配置：
export KAFKA_OPTS="-Djava.security.krb5.conf=/usr/local/kafka/config/kerberos/krb5.conf -Djava.security.auth.login.config=/usr/local/kafka/config/kerberos/kafka_server_jaas.conf"
3.2.3、Kafka 客户端配置
该配置主要为了使用 bin/kafka-topics.sh、bin/kafka-console-consumer.sh、kafka-console-producer.sh 等命令。
A、新建 /usr/local/kafka/config/kerberos/client.properties 文件
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
B、新建 /usr/local/kafka/config/kerberos/kafka_client_jaas.conf 文件
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/usr/local/kafka/config/kerberos/client.keytab"
storeKey=true
useTicketCache=false
principal="client@ABC.COM";
};
C、复制并修改 kafka-topics.sh、kafka-console-consumer.sh、kafka-console-producer.sh 脚本
cp /usr/local/kafka/bin/kafka-topics.sh /usr/local/kafka/bin/kafka-topics-sasl.sh cp /usr/local/kafka/bin/kafka-console-consumer.sh /usr/local/kafka/bin/kafka-console-consumer-sasl.sh cp /usr/local/kafka/bin/kafka-console-producer.sh /usr/local/kafka/bin/kafka-console-producer-sasl.sh
倒数第二行增加如下配置：
export KAFKA_OPTS="-Djava.security.krb5.conf=/usr/local/kafka/config/kerberos/krb5.conf -Djava.security.auth.login.config=/usr/local/kafka/config/kerberos/kafka_client_jaas.conf"
3.3、启动并测试
3.3.1、启动kafka
kafka-server-start-sasl.sh -daemon /usr/local/kafka/config/kerberos/server.properties
启动完成后可以使用 bin/kafka-topics.sh、kafka-console-consumer-sasl.sh、kafka-console-producer-sasl.sh来测试：
3.3.2、 创建一个测试的Topic
kafka-topics-sasl.sh --create --topic topictsasl --bootstrap-server 192.16.30.124:9092 --partitions 1 --replication-factor 1 --command-config /usr/local/kafka/config/kerberos/client.properties
3.3.3、查看所有Topic
kafka-topics-sasl.sh --list --bootstrap-server 192.16.30.124:9092 --command-config /usr/local/kafka/config/kerberos/client.properties
3.3.4、详细查看指定Topic
kafka-topics-sasl.sh --describe --topic topicsasl --bootstrap-server 192.16.30.124:9092 --command-config /usr/local/kafka/config/kerberos/client.properties
3.3.5、生产消息
kafka-console-producer-sasl.sh --topic topicsasl --bootstrap-server 192.16.30.124:9092 --producer.config /usr/local/kafka/config/kerberos/client.properties
3.3.6、消费消息
kafka-console-consumer-sasl.sh --topic topicsasl --from-beginning --bootstrap-server 192.16.30.124:9092 --consumer.config /usr/local/kafka/config/kerberos/client.properties
4、常见报错
1、Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry
[2022-07-05 18:58:33,590] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
at org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:69)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:168)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at kafka.server.BrokerToControllerChannelManagerImpl.newRequestThread(BrokerToControllerChannelManager.scala:189)
at kafka.server.BrokerToControllerChannelManagerImpl.<init>(BrokerToControllerChannelManager.scala:168)
at kafka.server.BrokerToControllerChannelManager$.apply(BrokerToControllerChannelManager.scala:131)
at kafka.server.KafkaServer.startup(KafkaServer.scala:276)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
解决办法：没有使用正确的kafka-server-start-sasl.sh文件启动，里面需要加上export KAFKA_OPTS才行
2、Krb认证报错 KrbException: Message stream modified (41)
[2022-07-07 16:23:51,436] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Message stream modified (41) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:97) at kafka.network.Processor.<init>(SocketServer.scala:724) at kafka.network.SocketServer.newProcessor(SocketServer.scala:367) at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:252) at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:251) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:214) at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:211) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:211) at kafka.network.SocketServer.startup(SocketServer.scala:122) at kafka.server.KafkaServer.startup(KafkaServer.scala:266) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:82) at kafka.Kafka.main(Kafka.scala) Caused by: javax.security.auth.login.LoginException: Message stream modified (41) at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:789) at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597) at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679) at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677) at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60) at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103) at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62) at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:112) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:158) ... 17 more Caused by: KrbException: Message stream modified (41) at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:103) at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159) at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310) at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498) at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:761)
解决办法：vim /etc/krb5.conf
删除 krb5.conf 配置文件里的 renew_lifetime = xxx 这行配置即可
