Helm 部署 Kafka：9093 SSL + 9092 PLAINTEXT 双端口(域名)集群外部访问

# Helm 部署 Kafka：9093 SSL + 9092 PLAINTEXT 双端口外部访问

> 适用于 Bitnami Kafka 3.9.x (KRaft 模式)，EKS + AWS NLB 内网场景。

---

## 架构概览

```
外部客户端 (EC2)
    │
    ├── kafka-datacenter-qa-N.icbc.com:9092  ──► NLB (internal) ──► Pod:9092  CLIENT (PLAINTEXT)
    └── kafka-datacenter-qa-N.icbc.com:9093  ──► NLB (internal) ──► Pod:9093  EXTERNAL (SSL)
                                                                        │
                                          集群内 broker 互联 (9095)  ◄──┘
```

| 监听器名称 | 端口 | 协议 | 用途 |
|-----------|------|------|------|
| CLIENT    | 9092 | PLAINTEXT | 外部客户端非加密访问 |
| EXTERNAL  | 9093 | SSL       | 外部客户端 TLS 加密访问 |
| INTERNAL  | 9095 | PLAINTEXT | Broker 间副本同步（集群内） |
| CONTROLLER| 9094 | PLAINTEXT | KRaft Controller 选举 |

---

## 前置条件

### 1. StorageClass

```bash
kubectl get storageclass kafka-gp3
```

需要存在支持 `WaitForFirstConsumer` 的 gp3 StorageClass。

### 2. TLS 证书 Secret

```bash
# 证书文件：tls.crt（含证书链）、tls.key、ca.crt
kubectl create secret generic kafka-datacenter-qa-tls \
  --from-file=tls.crt=./kafka.crt \
  --from-file=tls.key=./kafka.key \
  --from-file=ca.crt=./ca.crt \
  -n kafka-datacenter-qa
```

### 3. DNS 解析

每个 Pod 对应一个域名，需在 Route53（或 DNS 服务商）创建指向 NLB 的记录：

```
kafka-datacenter-qa-0.icbc.com →  NLB DNS
kafka-datacenter-qa-1.icbc.com →  NLB DNS
kafka-datacenter-qa-2.icbc.com →  NLB DNS
```

NLB 的 DNS 在 `externalAccess.controller.service.loadBalancerNames` 中预先指定，
AWS Load Balancer Controller 会自动创建对应名称的 NLB。

---

## values.yaml 核心配置说明

### 监听器定义

```yaml
listeners:
  client:
    protocol: PLAINTEXT       # 9092，不加密
  interbroker:
    protocol: PLAINTEXT
    containerPort: 9095       # broker 内部副本同步
  controller:
    protocol: PLAINTEXT
    containerPort: 9094       # KRaft controller
  external:
    protocol: SSL             # 9093，TLS 加密
    containerPort: 9093

tls:
  type: PEM
  existingSecret: kafka-datacenter-qa-tls   # 提前创建好的证书 Secret
```

### NLB 外部访问

```yaml
externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false            # 手动指定域名，不用 RBAC 自发现
  controller:
    service:
      type: LoadBalancer
      ports:
        external: 9093        # SSL 主端口
      extraPorts:
        - name: client-plaintext
          port: 9092
          targetPort: 9092
          protocol: TCP       # 在同一个 NLB 上额外暴露 9092
      loadBalancerNames:      # 每个 Pod 独占一个 NLB（按 ordinal 顺序对应）
        - kafka-datacenter-qa-0.licbc.com
        - kafka-datacenter-qa-1.licbc.com
        - kafka-datacenter-qa-2.licbc.com
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "external"
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"  # 内网 NLB
```

### advertised.listeners 补丁（核心难点）

**问题根源**：Bitnami `setup.sh` 在初始化时会将 CLIENT listener 的 advertised 地址写为
集群内 headless FQDN（`kafka-cluster-controller-N.kafka-cluster-controller-headless...`），
外部客户端无法解析这个地址，导致 bootstrap 成功但元数据返回后连接失败。

**解法**：在自定义 `command` 中，调用 `setup.sh` 之后直接覆写 `server.properties` 中的
`advertised.listeners` 行。

**关键坑**：
1. `/opt/bitnami/kafka/config/` 目录**只读**（无法创建新文件），但目录内已有文件可写。
   - ❌ `mv /tmp/xxx server.properties`：`/tmp` 是 tmpfs，跨设备 mv 失败
   - ❌ `> server.properties.new`：目录只读，无法创建新文件
   - ✅ 把内容读入 shell 变量，再用 `>` 直接覆写已有文件（无需创建新文件）

2. `export KAFKA_CFG_ADVERTISED_LISTENERS` 必须在 `setup.sh` 之前设置，但 `setup.sh`
   会在内部重新构造该变量，最终值仍然不对。因此必须在 `setup.sh` 之后再次覆写文件。

```yaml
controller:
  extraEnvVars:
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name

  command:
    - /bin/bash
    - -c
    - |
      ORDINAL="${MY_POD_NAME##*-}"
      DOMAIN="kafka-datacenter-qa-${ORDINAL}.icbc.com"
      INTERNAL_FQDN="kafka-cluster-controller-${ORDINAL}.kafka-cluster-controller-headless.kafka-datacenter-qa.svc.cluster.local"
      CONF_FILE="/opt/bitnami/kafka/config/server.properties"

      CORRECT_LISTENERS="CLIENT://${DOMAIN}:9092,INTERNAL://${INTERNAL_FQDN}:9095,EXTERNAL://${DOMAIN}:9093"
      export KAFKA_CFG_ADVERTISED_LISTENERS="${CORRECT_LISTENERS}"

      /opt/bitnami/scripts/kafka/setup.sh

      # 读入变量后直接覆写已有文件，目录只读但文件本身可写
      FILTERED=$(grep -v '^advertised\.listeners=' "${CONF_FILE}")
      printf '%s\n' "${FILTERED}" > "${CONF_FILE}"
      echo "advertised.listeners=${CORRECT_LISTENERS}" >> "${CONF_FILE}"

      exec /opt/bitnami/kafka/bin/kafka-server-start.sh ${CONF_FILE}
```

---

## 完整 values.yaml

```yaml
global:
  storageClass: "kafka-gp3"
  security:
    allowInsecureImages: true

image:
  registry: <YOUR_ECR_REGISTRY>
  repository: sretools
  tag: kafka-3.9.0
  pullPolicy: Always

controller:
  replicaCount: 3
  automountServiceAccountToken: true

  nodeSelector:
    topology.kubernetes.io/zone: ap-northeast-1a

  resources:
    requests:
      cpu: "200m"
      memory: "6Gi"
    limits:
      cpu: "2"
      memory: "8Gi"

  persistence:
    enabled: true
    size: 300Gi
    storageClass: "kafka-gp3"

  heapOpts: "-Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=16m -XX:SoftMaxHeapSize=2560m -XX:MaxDirectMemorySize=512m -XX:+G1UseAdaptiveIHOP -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -Xlog:gc*:stdout:time,tags"

  containerPorts:
    client: 9092
    controller: 9094

  extraEnvVars:
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name

  command:
    - /bin/bash
    - -c
    - |
      ORDINAL="${MY_POD_NAME##*-}"
      DOMAIN="kafka-datacenter-qa-${ORDINAL}.icbc.com"
      INTERNAL_FQDN="kafka-cluster-controller-${ORDINAL}.kafka-cluster-controller-headless.kafka-datacenter-qa.svc.cluster.local"
      CONF_FILE="/opt/bitnami/kafka/config/server.properties"

      CORRECT_LISTENERS="CLIENT://${DOMAIN}:9092,INTERNAL://${INTERNAL_FQDN}:9095,EXTERNAL://${DOMAIN}:9093"
      export KAFKA_CFG_ADVERTISED_LISTENERS="${CORRECT_LISTENERS}"

      /opt/bitnami/scripts/kafka/setup.sh

      FILTERED=$(grep -v '^advertised\.listeners=' "${CONF_FILE}")
      printf '%s\n' "${FILTERED}" > "${CONF_FILE}"
      echo "advertised.listeners=${CORRECT_LISTENERS}" >> "${CONF_FILE}"

      exec /opt/bitnami/kafka/bin/kafka-server-start.sh ${CONF_FILE}

broker:
  replicaCount: 0
  automountServiceAccountToken: true

listeners:
  client:
    protocol: PLAINTEXT
  interbroker:
    protocol: PLAINTEXT
    containerPort: 9095
  controller:
    protocol: PLAINTEXT
    containerPort: 9094
  external:
    protocol: SSL
    containerPort: 9093

tls:
  type: PEM
  existingSecret: kafka-datacenter-qa-tls

externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false
  controller:
    service:
      type: LoadBalancer
      ports:
        external: 9093
      extraPorts:
        - name: client-plaintext
          port: 9092
          targetPort: 9092
          protocol: TCP
      loadBalancerNames:
        - kafka-datacenter-qa-0.licbc.com
        - kafka-datacenter-qa-1.licbc.com
        - kafka-datacenter-qa-2.licbc.com
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "external"
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"

serviceAccount:
  create: true

service:
  type: ClusterIP
  ports:
    client: 9092
    controller: 9094
    interbroker: 9095

extraConfig: |
  num.network.threads=2
  num.io.threads=8
  socket.send.buffer.bytes=-1
  socket.receive.buffer.bytes=-1
  socket.request.max.bytes=157286400
  queued.max.requests=1000
  unclean.leader.election.enable=false
  replica.lag.time.max.ms=45000
  replica.fetch.max.bytes=16777216
  num.partitions=3
  default.replication.factor=2
  min.insync.replicas=2
  offsets.topic.replication.factor=3
  transaction.state.log.replication.factor=3
  transaction.state.log.min.isr=2
  num.recovery.threads.per.data.dir=1
  log.retention.check.interval.ms=1800000
  auto.leader.rebalance.enable=false
  group.initial.rebalance.delay.ms=5000
  compression.type=producer
  log.retention.hours=72
  log.segment.bytes=268435456
  log.cleanup.policy=delete
  auto.create.topics.enable=false
  delete.topic.enable=true
  controlled.shutdown.enable=true
  ssl.client.auth=none

sasl:
  client:
    users: []

rbac:
  create: true

metrics:
  jmx:
    enabled: true
    kafkaJmxPort: 9999
  kafka:
    enabled: true
  serviceMonitor:
    enabled: true
    namespace: "kafka-datacenter-qa"
    labels:
      release: kube-prom-stack
```

---

## 部署流程

```bash
# 1. 创建 namespace
kubectl create namespace kafka-datacenter-qa

# 2. 创建 TLS Secret
kubectl create secret generic kafka-datacenter-qa-tls \
  --from-file=tls.crt=./kafka.crt \
  --from-file=tls.key=./kafka.key \
  --from-file=ca.crt=./ca.crt \
  -n kafka-datacenter-qa

# 3. 部署
helm upgrade --install kafka-cluster \
  oci://registry-1.docker.io/bitnamicharts/kafka \
  --version 31.x.x \
  -n kafka-datacenter-qa \
  -f values.yaml

# 4. 等待 StatefulSet 就绪
kubectl rollout status statefulset -n kafka-datacenter-qa kafka-cluster-controller

# 5. 验证 advertised.listeners 是否正确
kubectl logs -n kafka-datacenter-qa kafka-cluster-controller-0 | grep '\[final\]'
kubectl logs -n kafka-datacenter-qa kafka-cluster-controller-1 | grep '\[final\]'
kubectl logs -n kafka-datacenter-qa kafka-cluster-controller-2 | grep '\[final\]'
```

预期输出：
```
[final] advertised.listeners=CLIENT://kafka-datacenter-qa-0.licbc.com:9092,INTERNAL://...headless...:9095,EXTERNAL://kafka-datacenter-qa-0.licbc.com:9093
```

---

## 验证连接

### PLAINTEXT 9092

```bash
kafka-topics.sh \
  --bootstrap-server kafka-datacenter-qa-0.icbc.com:9092 \
  --list
```

### SSL 9093

```bash
kafka-topics.sh \
  --bootstrap-server kafka-datacenter-qa-0.licbc.com:9093 \
  --command-config client-ssl.properties \
  --list
```

`client-ssl.properties` 示例：
```properties
security.protocol=SSL
ssl.truststore.location=/path/to/kafka.truststore.jks
ssl.truststore.password=changeit
```

### kcat 快速验证

```bash
# PLAINTEXT
kcat -b kafka-datacenter-qa-0.icbc.com:9092 -L

# SSL
kcat -b kafka-datacenter-qa-0.icbc.com:9093 \
  -X security.protocol=SSL \
  -X ssl.ca.location=/path/to/ca.crt \
  -L
```

---

## 故障排查

### 症状：bootstrap 成功但 9092 连接失败（UnknownHostException）

```
Error connecting to node kafka-cluster-controller-N.kafka-cluster-controller-headless...svc.cluster.local:9092
```

**原因**：`advertised.listeners` 中 CLIENT 地址仍为集群内 headless FQDN，外部客户端无法解析。

**排查**：
```bash
kubectl logs -n kafka-datacenter-qa kafka-cluster-controller-2 | grep '\[final\]'
kubectl exec -n kafka-datacenter-qa kafka-cluster-controller-2 -- \
  grep 'advertised.listeners' /opt/bitnami/kafka/config/server.properties
```

**修复**：确认 `command` 中的文件覆写步骤执行成功（无 `Read-only file system` 报错）。

### 症状：command 中无法创建临时文件

```
/bin/bash: /opt/bitnami/kafka/config/server.properties.new: Read-only file system
```

**原因**：`/opt/bitnami/kafka/config/` 目录本身只读，无法创建新文件，但已有文件可写。

**错误方案**：
- `mv /tmp/xxx ${CONF_FILE}` — `/tmp` 是 tmpfs，跨设备 mv 报 `Invalid cross-device link`
- `> ${CONF_FILE}.new` — 创建新文件报 `Read-only file system`

**正确方案**：先读入变量，再用 `>` 覆写已有文件：
```bash
FILTERED=$(grep -v '^advertised\.listeners=' "${CONF_FILE}")
printf '%s\n' "${FILTERED}" > "${CONF_FILE}"
echo "advertised.listeners=${CORRECT_LISTENERS}" >> "${CONF_FILE}"
```

### 症状：setup.sh 调用后 KAFKA_CFG_ADVERTISED_LISTENERS 被覆盖

`export KAFKA_CFG_ADVERTISED_LISTENERS` 在 `setup.sh` 之前设置无效，因为 `setup.sh`
会根据 chart 配置重新生成该值。必须在 `setup.sh` 完成后再修改 `server.properties`。

---

## 注意事项

- `ssl.client.auth=none`：9093 只做传输加密，不要求客户端提供证书（单向 TLS）。
  如需双向认证，改为 `ssl.client.auth=required`。
- NLB 按 Pod ordinal 顺序一一对应，`loadBalancerNames` 列表顺序不能打乱。
- `autoDiscovery.enabled: false` 时无需给 ServiceAccount 赋予 `list services` 权限。
- KRaft 模式下 `broker.replicaCount: 0`，controller 节点同时承担 broker 角色。