kubernetes的存储机制基于StorageClass实现NFS的动态置备nfs-subdir-external-provisioner项目案例

storageClass

1）分类属性字段,静态置备（手动创建PV）
2）storageClass独立的资源使用，动态置备（自动创建PV）

Pod --> PVC--> PV

NFS StorageClass

NFS的存储制备器方案

NFS 的自动配置程序 Provisioner 可以通过不同的项目实现,比如

https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes/#nfs

NFS-Subdir-External-Provisioner
此组件是由Kubernetes SIGs 社区开发,也是Kubernetes官方推荐实现是对 nfs-client-provisioner 组件的扩展
https://kubernetes.io/docs/concepts/storage/storage-classes/#nfs

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner

NFS-Subdir-External-Provisioner 是一个自动配置卷程序，可以在 NFS 服务器上通过PVC动态创建和配置 Kubernetes 持久卷
PV命名规则如下

自动创建的 PV 以${namespace}-${pvcName}-${pvName} 命名格式创建在 NFS 服务器上的共享数据目录中
当这个 PV 被回收后会以 archieved-${namespace}-${pvcName}-${pvName} 命名格式存在NFS 服务器中

案例: 基于 nfs-subdir-external-provisioner 创建 NFS 共享存储的storageclass

部署相关文件

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
https://github.com/kubernetes-sigs/nfs-subdir-externalprovisioner/tree/master/deploy
https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner?tab=readme-ovfile#manually

创建 NFS 共享存储的 storageclass步骤如下：

创建 NFS 共享
创建 Service Account 并授予管控NFS provisioner在k8s集群中运行的权限
部署 NFS-Subdir-External-Provisioner 对应的 Deployment
创建 StorageClass 负责建立PVC并调用NFS provisioner进行预定的工作,并让PV与PVC建立联系
创建 PVC 时自动调用SC创建PV
创建Pod 使用 PVC

创建 NFS 服务

#NFS服务器软件安装

# 安装 nfs-utils（包含 nfs-server 服务）
dnf install -y nfs-utils

# 启动并设置开机自启
systemctl enable --now nfs-server

# 验证状态
systemctl status nfs-server

mkdir -p /data/sc-nfs/

#授权worker节点的网段可以挂载

[root@master1 ~]# cat /etc/exports
#/data/nfsshare  192.168.3.0/24(rw,sync,no_root_squash)
#/nfsdata *(rw,no_root_squash)
/data/sc-nfs *(rw,no_root_squash)

[root@master1 ~]# exportfs -r
[root@master1 ~]# exportfs -v
/data/sc-nfs    <world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

#并在所有worker节点安装NFS客户端

dnf install -y nfs-utils

#在所有节点节点添加nfs..org的域名解析指定NFS服务器的地址

[root@node1 ~]# 
[root@node1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.3.60 kubeapi.org kubeapi
192.168.3.60 master1.org master1
192.168.3.60 nfs.org master1
192.168.3.61 master2.org master2
192.168.3.62 master3.org master3
192.168.3.63 node1.org node1
192.168.3.64 node2.org node2
192.168.3.65 node3.org node3
192.168.3.66 ha1.org ha1
192.168.3.67 ha2.org ha2

创建 ServiceAccount 并授权

#建议创建独立的名称空间

kubectl create ns sc-nfs

[root@master1 sc-nfs]# cat rbac.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  #namespace: default
  namespace: sc-nfs
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    #namespace: default
    namespace: sc-nfs
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  #namespace: default
  namespace: sc-nfs
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  #namespace: default
  namespace: sc-nfs
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    #namespace: default
    namespace: sc-nfs
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f rbac.yaml

[root@master1 sc-nfs]# kubectl get sa -n sc-nfs
NAME                     AGE
default                  4m8s
nfs-client-provisioner   46s

部署 NFS-Subdir-External-Provisioner 对应的 Deployment

[root@master1 sc-nfs]# cat nfs-client-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  #namespace: default
  namespace: sc-nfs
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-beijing.aliyuncs.com/wangxiaochun/nfs-subdir-external-provisioner:v4.0.2
          #image: wangxiaochun/nfs-subdir-external-provisioner:v4.0.2
          #image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner #名称确保与 nfs-StorageClass.yaml文件中的provisioner名称保持一致
            - name: NFS_SERVER
              value: nfs.org # NFS SERVER_IP 
            - name: NFS_PATH
              value: /data/sc-nfs  # NFS 共享目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: nfs.org  # NFS SERVER_IP 
            path: /data/sc-nfs  # NFS 共享目录

通过第三方软件管理实现nfs动态置备功能

需要设置环境变量

通过第三方软件管理nfs服务器 /data/sc-nfs下的子目录每个子目录自动创建对应pv

kubectl apply -f nfs-client-provisioner.yaml

[root@master1 sc-nfs]# kubectl get pod -n sc-nfs -o wide
NAME                                     READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
nfs-client-provisioner-d5dc647db-w8gxw   1/1     Running   0          28s   10.244.2.187   node2.org   <none>           <none>

#注意:如果失败,检查是否worker节点安装了nfs-client

创建 NFS 资源的 StorageClass

StorageClass 作用声明使用第三方置备软件管理pv

root@master1 sc-nfs]# cat nfs-StorageClass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sc-nfs 
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"  # 是否设置为默认的storageclass
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true" # 设置为"false"时删除PVC不会保留数据,"true"则保留数据
[root@master1 sc-nfs]#

#应用

kubectl apply -f nfs-StorageClass.yaml

#查看是否创建

[root@master1 sc-nfs]# kubectl get sc -n sc-nfs   
NAME            PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage   kubernetes.io/no-provisioner                  Delete          WaitForFirstConsumer   false                  14h
sc-nfs          k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate              false                  18s

创建 PVC

[root@master1 sc-nfs]# cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-sc
spec:
  storageClassName: sc-nfs  #需要和前面创建的storageClass名称相同
  accessModes: ["ReadWriteMany","ReadOnlyMany"]
  resources:
    requests:
      storage: 100Mi

[root@master1 sc-nfs]# kubectl get pv          
No resources found
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# kubectl get pvc
No resources found in default namespace.
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# ls /data/sc-nfs/

kubectl apply -f pvc.yaml

#验证

kubectl get pvc

[root@master1 sc-nfs]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-nfs-sc   Bound    pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca   100Mi      ROX,RWX        sc-nfs         <unset>                 4s

#自动创建PV并绑定PVC

kubectl get pv

[root@master1 sc-nfs]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca   100Mi      ROX,RWX        Delete           Bound    default/pvc-nfs-sc   sc-nfs         <unset>                          20s

root@master1 sc-nfs]# ls /data/sc-nfs/
default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca

命名空间+pvc名字+pv名字

default+pvc-nfs-sc+pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca

[root@master1 sc-nfs]# ls /data/sc-nfs/default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca/
[root@master1 sc-nfs]#

创建 Pod

[root@master1 sc-nfs]# cat pod-test.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-nfs-sc-test
spec:
  containers:
  - name: pod-nfs-sc-test
    image: registry.cn-beijing.aliyuncs.com/wangxiaochun/nginx:1.20.0
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/usr/share/nginx/html/"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: pvc-nfs-sc  #指定前面创建的PVC名称

#应用

kubectl apply -f pod-test.yaml

#验证

[root@master1 sc-nfs]# kubectl get pod -o wide
NAME                                        READY   STATUS    RESTARTS       AGE     IP             NODE        NOMINATED NODE   READINESS GATES
controller-deployment-demo-5f884bb8-2ppvf   1/1     Running   5 (50m ago)    40h     10.244.1.192   node1.org   <none>           <none>
controller-deployment-demo-5f884bb8-fscrl   1/1     Running   14 (50m ago)   7d20h   10.244.1.193   node1.org   <none>           <none>
controller-deployment-demo-5f884bb8-rfsbv   1/1     Running   13 (50m ago)   7d16h   10.244.1.194   node1.org   <none>           <none>
pod-nfs-sc-test                             1/1     Running   0              5s      10.244.2.188   node2.org   <none>           <none>

# 查看nginx的网页文件不存在

#由于没有页面文件,访问Pod失败

[root@master1 sc-nfs]# curl 10.244.2.188
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.20.0</center>
</body>
</html>
[root@master1 sc-nfs]# ls /data/sc-nfs/default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca/

#在NFS服务器创建页面文件

echo "NFS-SC Website" >> /data/sc-nfs/default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca/index.html

#查看Pod内的文件

[root@master1 sc-nfs]# kubectl exec -it pod-nfs-sc-test -- ls /usr/share/nginx/html
index.html

#访问Pod页面成功

root@master1 sc-nfs]# curl 10.244.2.188      
NFS-SC Website

没有指定storageClassName 名字创建pvc会Pending

root@master1 sc-nfs]# cat pvc2.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-sc2
spec:
  #storageClassName: sc-nfs  #需要和前面创建的storageClass名称相同
  accessModes: ["ReadOnlyMany"]
  resources:
    requests:
      storage: 10Mi
[root@master1 sc-nfs]#

kubectl apply -f pvc2.yaml

[root@master1 sc-nfs]# kubectl get pvc
NAME          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-nfs-sc    Bound     pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca   100Mi      ROX,RWX        sc-nfs         <unset>                 17m
pvc-nfs-sc2   Pending

指定storageClassName 名字，或者指定storageClass 默认值为true 默认用sc-nfs为storageClassName 名字

[root@master1 sc-nfs]# cat nfs-StorageClass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sc-nfs 
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"  # 是否设置为默认的storageclass
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true" # 设置为"false"时删除PVC不会保留数据,"true"则保留数据

kubectl apply -f nfs-StorageClass.yaml

[root@master1 sc-nfs]# kubectl get sc
NAME               PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
sc-nfs (default)   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  30m

绑定上去 pv也自动创建好

[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# kubectl get pvc            
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-nfs-sc    Bound    pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca   100Mi      ROX,RWX        sc-nfs         <unset>                 21m
pvc-nfs-sc2   Bound    pvc-5c72627d-d450-4c24-be17-d854c427694e   10Mi       ROX            sc-nfs         <unset>                 4m34s
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-5c72627d-d450-4c24-be17-d854c427694e   10Mi       ROX            Delete           Bound    default/pvc-nfs-sc2   sc-nfs         <unset>                          11s
pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca   100Mi      ROX,RWX        Delete           Bound    default/pvc-nfs-sc    sc-nfs         <unset>                          21m
[root@master1 sc-nfs]#

[root@master1 sc-nfs]# ls /data/sc-nfs/
default-pvc-nfs-sc2-pvc-5c72627d-d450-4c24-be17-d854c427694e  default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca

删除 Pod 和 PVC

#先删除Pod

kubectl delete -f pod-test.yaml

#再删除PVC

kubectl delete -f pvc.yaml

#自动删除对应的PV

[root@master1 sc-nfs]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-5c72627d-d450-4c24-be17-d854c427694e   10Mi       ROX            Delete           Bound    default/pvc-nfs-sc2   sc-nfs         <unset>                          4m29s
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-nfs-sc2   Bound    pvc-5c72627d-d450-4c24-be17-d854c427694e   10Mi       ROX            sc-nfs         <unset>                 8m57s

#查看对应的数据不会删除,仍保留

[root@master1 sc-nfs]# ls /data/sc-nfs/archived-default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167eaca/
index.html

[root@master1 sc-nfs]# kubectl delete pvc pvc-nfs-sc2
persistentvolumeclaim "pvc-nfs-sc2" deleted from default namespace
[root@master1 sc-nfs]# kubectl get pv
No resources found
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# ls /data/sc-nfs/
archived-default-pvc-nfs-sc2-pvc-5c72627d-d450-4c24-be17-d854c427694e  archived-default-pvc-nfs-sc-pvc-7e0e1845-0edd-4e25-ba23-4079c167ea

基于sc-nfs 实现MySQL的动态置备

[root@master1 sc-nfs]# cat storage-mysql-storage-class-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: sc-nfs
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: registry.cn-beijing.aliyuncs.com/wangxiaochun/mysql:8.0.29-oracle
        name: mysql
        env:
          # 在实际中使用 secret
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

kubectl apply -f storage-mysql-storage-class-pvc.yaml

[root@master1 sc-nfs]# kubectl get pod
NAME                                        READY   STATUS    RESTARTS       AGE
controller-deployment-demo-5f884bb8-2ppvf   1/1     Running   5 (76m ago)    41h
controller-deployment-demo-5f884bb8-fscrl   1/1     Running   14 (76m ago)   7d20h
controller-deployment-demo-5f884bb8-rfsbv   1/1     Running   13 (76m ago)   7d17h
mysql-66694bb69b-7hqbh                      1/1     Running   0              25s
[root@master1 sc-nfs]# 
[root@master1 sc-nfs]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-95692727-c5b5-4bec-b212-af8864bd69a6   20Gi       RWO            Delete           Bound    default/mysql-pv-claim   sc-nfs         <unset>                          27s
[root@master1 sc-nfs]# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
mysql-pv-claim   Bound    pvc-95692727-c5b5-4bec-b212-af8864bd69a6   20Gi       RWO            sc-nfs         <unset>                 29s

[root@master1 sc-nfs]# ls /data/sc-nfs/default-mysql-pv-claim-pvc-95692727-c5b5-4bec-b212-af8864bd69a6/
 auto.cnf        binlog.index   client-cert.pem     '#ib_16384_1.dblwr'   ib_logfile0  '#innodb_temp'   mysql.sock           public_key.pem    sys
 binlog.000001   ca-key.pem     client-key.pem       ib_buffer_pool       ib_logfile1   mysql           performance_schema   server-cert.pem   undo_001
 binlog.000002   ca.pem        '#ib_16384_0.dblwr'   ibdata1              ibtmp1        mysql.ibd       private_key.pem      server-key.pem    undo_002

posted @ 2026-05-24 16:43 minger_lcm 阅读(25) 评论(0) 收藏举报

刷新页面返回顶部

kubernetes的存储机制基于StorageClass实现NFS的动态置备nfs-subdir-external-provisioner项目案例

storageClass

公告