K8s部署Apache-Nifi集群（下）
本文为博主独立完成的原创技术文档，首发于个人博客。
转载须知：
✅ 欢迎分享本文链接至技术群、朋友圈
✅ 个人收藏、学习使用不限
❌ 严禁洗稿、抄袭，或未经授权的商业转载/汇编
📩 如需转载至其他平台，请私信联系博主获取授权
作者：赵俊杰
一.概述
见上篇文章
二.Nifi部署（上）
见上篇文章
三.Nifi部署（下）
创建Nifi集群的pv资源清单:
[root@master231 nifi-cluster]# cat nifi-pv-cluster.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nifi-cluster-1 #nifi集群的pv名称
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /data/nifi-cluster #指定nifi集群的存储目录（需要在对应的节点提前创建目录）
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- master231 #指定调度到master231节点上
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nifi-cluster-2
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /data/nifi-cluster
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- worker232
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nifi-cluster-3
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /data/nifi-cluster
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- worker233
创建目录：需要在master231，worker232，worker233全部准备
[root@master231 ~]# mkdir -p /data/nifi-cluster/
[root@worker232 ~]# mkdir -p /data/nifi-cluster/
[root@worker233 ~]# mkdir -p /data/nifi-cluster/
（此时的路径需要与nifi集群的pv资源清单中路径一致）
创建Nifi集群的secret资源清单：
进入到证书目录
[root@master231 nifi-cluster]# cd cert
[root@master231 cert]# cat nifi-certs-secrets.yml
apiVersion: v1
kind: Secret
metadata:
name: nifi-server-keystore #指定secret的名称
namespace: apache-nifi-cluster #指定名称空间
type: Opaque
data:
keystore.p12: 这里要写keystore.p12.base64中的内容编码（自定义证书目录中的文件）
---
apiVersion: v1
kind: Secret
metadata:
name: nifi-truststore
namespace: apache-nifi-cluster
type: Opaque
data:
truststore.p12: 这里要写truststore.p12.base64中的内容编码（自定义证书目录中的文件）
创建Nifi集群的sts资源清单：
[root@master231 cert]# cd ..
[root@master231 nifi-cluster]#
[root@master231 nifi-cluster]# cat nifi-cluster.yml
apiVersion: apps/v1
kind: StatefulSet #适用于有状态的服务
metadata:
name: nifi
namespace: apache-nifi-cluster
labels:
app: nifi #标签为nifi
spec:
serviceName: nifi-headless #指定与这个 StatefulSet关联的Headless Service名称 为 nifi-headless。无头服务用于为每个Pod提供唯一的网络标识。
replicas: 3
selector:
matchLabels:
app: nifi
template:
metadata:
labels:
app: nifi
spec:
subdomain: nifi-headless
dnsConfig:
searches:
- nifi-headless.apache-nifi-cluster.svc.1113v.com #指定DNS搜索域，组成为nifi-headless和名称空间和dns解析名称（k8s的名称）
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: nifi
topologyKey: "kubernetes.io/hostname" #定义了 Pod 的反亲和性规则，确保带有app: nifi 标签的 Pod 不会调度到同一个节点上
hostAliases: #定义了 Pod 的 /etc/hosts 文件中的主机别名，将 IP 地址映射到主机名
- ip: "10.131.10.144"
hostnames:
- "es-zb"
- ip: "10.131.10.49"
hostnames:
- "elknode1"
- ip: "10.131.10.51"
hostnames:
- "elknode3"
- ip: "10.131.10.50"
hostnames:
- "elknode2"
initContainers: #定义初始化容器，创建所需nifi目录
- name: init-set-permissions
image: busybox:latest
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "mkdir -p /data/database_repository && mkdir -p /data/content_repository && mkdir -p /data/provenance_repository && mkdir -p /data/flowfile_repository && mkdir -p /data/conf && chmod 755 /data/* && chown 1000:1000 /data/*"]
volumeMounts:
- name: nifi-storage
mountPath: /data
- name: copy-conf
image: apache/nifi:2.0.0
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
if [ -f /data/conf/nifi.properties ]; then
echo "Conf directory already exists, skipping copy."
else
echo "Conf directory does not exist, copying files..."
cp -r /opt/nifi/nifi-current/conf/* /data/conf
fi
volumeMounts:
- name: nifi-storage
mountPath: /data
containers:
- name: nifi
image: apache/nifi:2.0.0
ports:
- containerPort: 8443
name: https
env:
- name: TZ
value: "Asia/Shanghai"
- name: NIFI_CLUSTER_IS_NODE
value: "true"
- name: METADATA_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NIFI_CLUSTER_ADDRESS
value: "$(METADATA_NAME).nifi-headless.apache-nifi-cluster.svc.1113v.com"
- name: NIFI_CLUSTER_NODE_PROTOCOL_PORT
value: "6007"
- name: NIFI_CLUSTER_NODE_PROTOCOL_MAX_THREADS
value: "20"
- name: NIFI_ZK_CONNECT_STRING
value: "zk-zookeeper-0.zk-zookeeper-headless.apache-nifi-cluster.svc.1113v.com:2181,zk-zookeeper-1.zk-zookeeper-headless.apache-nifi-cluster.svc.1113v.com:2181,zk-zookeeper-2.zk-zookeeper-headless.apache-nifi-cluster.svc.1113v.com:2181"
- name: NIFI_ZK_ROOT_NODE
value: "/nifi"
- name: NIFI_JVM_HEAP_INIT
value: "2g"
- name: NIFI_JVM_HEAP_MAX
value: "2g"
- name: NIFI_WEB_HTTP_HOST
value: "0.0.0.0"
- name: NIFI_WEB_HTTP_PORT
value: "8080"
- name: NIFI_WEB_PROXY_HOST
value: "10.0.0.231:30111"
- name: NIFI_WEB_HTTPS_HOST
value: "$(METADATA_NAME).nifi-headless"
- name: SINGLE_USER_CREDENTIALS_USERNAME
value: "admin"
- name: NIFI_SENSITIVE_PROPS_KEY
value: "yDGEwdsbdaQLJ5xdXpwuV62GyjQLIZTq"
- name: SINGLE_USER_CREDENTIALS_PASSWORD
value: "1234567890123"
- name: KEYSTORE_PATH
value: "./conf/keystore.p12"
- name: KEYSTORE_TYPE
value: "PKCS12"
- name: KEYSTORE_PASSWORD
value: "d081bfbc24efc1ea1a85839545b3b3b2"
- name: KEY_PASSWORD
value: "d081bfbc24efc1ea1a85839545b3b3b2"
- name: TRUSTSTORE_PATH
value: "./conf/truststore.p12"
- name: TRUSTSTORE_TYPE
value: "PKCS12"
- name: TRUSTSTORE_PASSWORD
value: "ec1452bfb681f7d44bdff1724c5c85a5"
- name: AUTH
value: "tls"
- name: NIFI_SECURITY_USER_LOGIN_IDENTITY_PROVIDER
value: "single-user-provider"
- name: NIFI_SECURITY_USER_AUTHORIZER
value: "single-user-authorizer"
securityContext:
runAsUser: 1000
runAsGroup: 1000
volumeMounts:
- name: nifi-storage
mountPath: /opt/nifi/nifi-current/database_repository
subPath: database_repository
- name: nifi-storage
mountPath: /opt/nifi/nifi-current/content_repository
subPath: content_repository
- name: nifi-storage
mountPath: /opt/nifi/nifi-current/provenance_repository
subPath: provenance_repository
- name: nifi-storage
mountPath: /opt/nifi/nifi-current/flowfile_repository
subPath: flowfile_repository
- name: nifi-storage
mountPath: /opt/nifi/nifi-current/conf
subPath: conf
- name: nifi-server-keystore
mountPath: /opt/nifi/nifi-current/conf/keystore.p12
subPath: keystore.p12
readOnly: true
- name: nifi-truststore
mountPath: /opt/nifi/nifi-current/conf/truststore.p12
subPath: truststore.p12
readOnly: true
volumes:
- name: nifi-server-keystore
secret:
secretName: nifi-server-keystore
- name: nifi-truststore
secret:
secretName: nifi-truststore
volumeClaimTemplates:
- metadata:
name: nifi-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi #需求大小和nifi集群的pv资源清单中的storage字段是匹配的，这里的需求大小必须小于nifi集群的pv资源中的大小，否则无法匹配成功。
storageClassName: local-storage
Nifi集群的svc资源清单：
[root@master231 nifi-cluster]# cat nifi-svc.yml
apiVersion: v1
kind: Service
metadata:
name: nifi-headless
namespace: apache-nifi-cluster
spec:
clusterIP: None #表示这是一个 Headless Service。与普通的 Svc不同，Headless Service 不会分配一个虚拟的集群 IP，客户端可以直接通过 DNS 解析到后端的 Pod IP 地址。
ports:
- name: http
port: 8443
#targetPort: http
targetPort: 8443
- name: cluster
port: 6007
#targetPort: cluster
targetPort: 6007
selector:
app: nifi
Ingress的资源清单:
[root@master231 nifi-cluster]# cat nifi-ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nifi-ingress
namespace: apache-nifi-cluster
annotations: #用于向 Ingress 控制器提供额外配置信息的键值对
nginx.ingress.kubernetes.io/affinity: "cookie"
# 启用基于 Cookie 的会话亲和性。这意味着同一个客户端的后续请求会被路由到同一个后端 Pod，有助于保持会话状态。
nginx.ingress.kubernetes.io/session-cookie-name: "NIFI_SESSION"
#指定用于会话亲和性的 Cookie 名称为 NIFI_SESSION。
nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
#指定用于生成会话cookie值的哈希算法为sha1
nginx.ingress.kubernetes.io/backend-ssl-verify: "false"
#禁用对后端服务ssl证书的验证。
nginx.ingress.kubernetes.io/backend-ca-secret: "ingress-backend-nifi-cert"
#指定用于验证后端服务SSL证书的CA证书所在的Secret名称为 ingress-backend-nifi-cert
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
#指定与后端服务通信使用的协议为 HTTPS
nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout http_502 http_503 http_504"
#当出现指定的错误时候，Ingress 控制器会尝试将请求转发到下一个可用的后端 Pod。
nginx.ingress.kubernetes.io/session-cookie-expires: "3600"
#指定会话 Cookie 的过期时间为 3600 秒（即 1 小时）。
nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
#指定会话 Cookie 的最大有效期为 3600 秒（即 1 小时）。
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
#启用 SSL 透传功能，即 Ingress 控制器不终止SSL连接，而是直接将加密的流量转发到后端服务
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
#指定 Ingress 控制器从后端服务读取响应的超时时间为 3600 秒。
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
#指定 Ingress 控制器向后端服务发送请求的超时时间为 3600 秒。
ingress.kubernetes.io/ssl-passthrough: "true"
#kubernetes.io/ingress.class: "nginx"
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nifi-headless
port:
number: 8443
部署nifi集群:
[root@master231 nifi-cluster]# kubectl apply -f nifi-pv-cluster.yml
[root@master231 nifi-cluster]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
nifi-cluster-1 3Gi RWO Retain Bound apache-nifi-cluster/nifi-storage-nifi-0 local-storage 11d
nifi-cluster-2 3Gi RWO Retain Bound apache-nifi-cluster/nifi-storage-nifi-1 local-storage 11d
nifi-cluster-3 3Gi RWO Retain Bound apache-nifi-cluster/nifi-storage-nifi-2 local-storage 11d
[root@master231 nifi-cluster]# kubectl apply -f nifi-cluster.yml
[root@master231 nifi-cluster]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nifi-0 1/1 Running 0 12s
nifi-1 1/1 Running 0 10s
nifi-2 1/1 Running 0 5s
zk-zookeeper-0 1/1 Running 0 25m
zk-zookeeper-1 1/1 Running 0 25m
zk-zookeeper-2 1/1 Running 0 25m
[root@master231 nifi-cluster]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nifi-0 1/1 Running 1 (17s ago) 42s
pod/nifi-1 1/1 Running 1 (11s ago) 40s
pod/nifi-2 1/1 Running 1 (6s ago) 35s
pod/zk-zookeeper-0 1/1 Running 0 25m
pod/zk-zookeeper-1 1/1 Running 0 25m
pod/zk-zookeeper-2 1/1 Running 0 25m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/zk-zookeeper ClusterIP 10.200.133.189 <none> 2181/TCP,2888/TCP,3888/TCP 25m
service/zk-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 25m
NAME READY AGE
statefulset.apps/nifi 3/3 42s
statefulset.apps/zk-zookeeper 3/3 25m
部署svc资源：
[root@master231 nifi-cluster]# kubectl apply -f nifi-svc.yml
service/nifi-headless created
查看svc的资源:
[root@master231 nifi-cluster]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nifi-headless ClusterIP None <none> 8443/TCP,6007/TCP 9s
zk-zookeeper ClusterIP 10.200.133.189 <none> 2181/TCP,2888/TCP,3888/TCP 27m
zk-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 27m
部署ingress的控制器资源:
创建helm安装的脚本
[root@master231 nifi-cluster]# cat helm-ingress-nodeport.sh
#!/bin/bash
helm install ingress-nginx ingress-nginx \
--set controller.service.type=NodePort \
--set controller.service.nodePorts.http=30111 \
--set controller.service.nodePorts.https=30443 \
--set controller.extraArgs.enable-ssl-passthrough=true \
--set controller.extraArgs.default-ssl-certificate=\$\(POD_NAMESPACE\)/ingress-backend-nifi-cert
[root@master231 nifi-cluster]# sh helm-ingress-nodeport.sh
查看控制器是否安装成功
[root@master231 nifi-cluster]# kubectl get pods
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-55b69f6586-rsmlj 1/1 Running 0 22s
nifi-0 1/1 Running 4 (4m16s ago) 6m17s
nifi-1 1/1 Running 4 (4m27s ago) 6m15s
nifi-2 1/1 Running 4 (4m22s ago) 6m10s
zk-zookeeper-0 1/1 Running 0 31m
zk-zookeeper-1 1/1 Running 0 31m
zk-zookeeper-2 1/1 Running 0 31m
部署ingress资源:
[root@master231 nifi-cluster]# kubectl apply -f nifi-ingress.yml
ingress.networking.k8s.io/nifi-ingress unchanged
[root@master231 nifi-cluster]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nifi-ingress nginx * 80 10s
部署 secret资源:
进入到证书目录，并创建secret资源
[root@master231 nifi-cluster]# cd cert
[root@master231 cert]# kubectl apply -f nifi-certs-secrets.yml
secret/nifi-server-keystore created
secret/nifi-truststore created
[root@master231 cert]# kubectl create secret tls ingress-backend-nifi-cert --key server.key --cert server.crt
secret/ingress-backend-nifi-cert created
查看secret资源
[root@master231 cert]# kubectl get secrets
NAME TYPE DATA AGE
default-token-vxkvv kubernetes.io/service-account-token 3 9d
ingress-backend-nifi-cert kubernetes.io/tls 2 47s
ingress-nginx-admission Opaque 3 3d23h
nifi-server-keystore Opaque 1 100s
nifi-truststore Opaque 1 100s
sh.helm.release.v1.zk.v1 helm.sh/release.v1 1 23m
zk-zookeeper-token-w55hc kubernetes.io/service-account-token 3 23m
访问界面进行测试：
访问网址
https://10.0.0.231:30443/nifi/#/login
查看nifi集群的证书是否为自定义证书，在访问界面的证书查看栏查看过期时间去验证
不必追赶别人的节奏，把普通的日子慢慢过好，就已经很厉害了。
本文来自博客园，作者：超级喜欢下雨天，转载请注明原文链接：https://www.cnblogs.com/zhao-junjie/p/22837069
浙公网安备 33010602011771号