local本地动态存储卷及pvc延迟绑定

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

目录

一.为什么要使用local卷

1.使用PV存储数据的痛点

- 1.基于网络存储的PV通常性能损耗较大，因为要涉及到跨节点传输数据，尤其是pod和pv持久卷不在同一个节点的情况下;

- 2.直接使用节点本地的SSD磁盘可获取较好的IO性能，更适用于存储类的服务，例如MongoDB、Ceph，MySQL，ElasticSearch等。

2.hostPath解决pv的问题

这个时候你可能会想到使用hostPath来解决上面提到的问题，但是我们不得不承认以下的问题:
	- 1.hostPath卷在Pod被重建后可能被调试至其它节点而无法再次使用此前的数据的情况;
	- 2.hostPath卷允许Pod访问节点上的任意路径，也存在一定程度的安全风险;

3.local卷解决hostPath存储卷的不足

local卷插件主要用于将本地存储设备（如磁盘、分区或目录）配置为卷。

local卷相比hostPath而言，带来了如下优势:
	- 1.基于local卷，调度器能自行完成调度绑定；
		说白了就是当第一次完成调度后，后续重新创建Pod时都会被调度到该节点，因为K8S调度器之前系统有记录。
		
	- 2.local卷只能关联静态置备的PV，目前尚不支持动态置备;
	
	- 3.local卷支持"延迟绑定"特性，延迟绑定机制，提供了基于消费者的需求来判定将PVC绑定至哪个PV的可能性             
		说白了，在Pod完成调度之前，还没有消费pvc时，此时的pvc一直处于Pending状态。
		举个例子: 
    		Pod有3个节点可以满足调度，但由于Pod没有完成调度前，pvc并不知道调度到那个pv，
    		因为pv是关联的是local卷，其必须和pod在同一个节点才行。

二.local卷案例

1.创建local动态存储类

	1.编写资源清单
[root@master231 yinzhengjie-k8s]# cat 01-local-sc.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: yinzhengjie-local-sc
# "kubernetes.io/no-provisioner"表示不使用动态置备PV，因为local插件不支持
provisioner: kubernetes.io/no-provisioner
# "WaitForFirstConsumer"表示等待消费者（Pod）申请使用PVC时（即第一次被调度时）再进行PV绑定，即“延迟绑定”
# 延迟绑定机制，提供了基于消费者的需求来判定将PVC绑定至哪个PV的可能性
# 说白了，在Pod完成调度之前，还没有消费pvc时，此时的pvc一直处于Pending状态。
# 举个例子: 
# 	  Pod有3个节点可以满足调度，但由于Pod没有完成调度前，pvc并不知道调度到那个pv，
#     因为pv是本地卷。其必须和pod在同一个节点才行。
volumeBindingMode: WaitForFirstConsumer

[root@master231 yinzhengjie-k8s]# 


	2.创建动态存储类
[root@master231 yinzhengjie-k8s]# kubectl apply -f 01-local-sc.yaml 
storageclass.storage.k8s.io/yinzhengjie-local-sc created
[root@master231 yinzhengjie-k8s]# 


	3.查看动态存储类（这个存储类无法自动创建pv，因为此配置为"kubernetes.io/no-provisioner"）
[root@master231 yinzhengjie-k8s]# kubectl get sc
NAME                   PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
yinzhengjie-local-sc   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  3s
[root@master231 yinzhengjie-k8s]#

2.手动创建pv关联sc

此处我故意创建两个pv均关联sc，且两个pv关联的本地卷在不同的节点和路径。


实战案例:
	1.编写资源清单
[root@master231 yinzhengjie-k8s]# cat 02-custom-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: yinzhengjie-pv-worker232
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  # 指定pv关联的存储类 
  storageClassName: yinzhengjie-local-sc
  # 表示卷插件类型是local
  local:
    # 指定本地的路径，这个路径最好是对应的是一块本地磁盘，此路径最好提前创建好，若路径不存在，将来调度成功后会报错:
    #	MountVolume.NewMounter initialization failed for volume "..." : path "..." does not exist
    path: /yinzhengjie/data/sc/local01
  # 声明pv与节点的关联关系，与此同时，调度器将基于nodeAffinity将影响Pod调度。
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker232

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: yinzhengjie-pv-worker233
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: yinzhengjie-local-sc
  local:
    path: /yinzhengjie/data/sc/local02
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker233
[root@master231 yinzhengjie-k8s]# 


	2.创建pv
[root@master231 yinzhengjie-k8s]# kubectl apply -f 02-custom-pv.yaml 
persistentvolume/yinzhengjie-pv-worker232 created
persistentvolume/yinzhengjie-pv-worker233 created
[root@master231 yinzhengjie-k8s]# 
[root@master231 yinzhengjie-k8s]# kubectl get pv 
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS           REASON   AGE
yinzhengjie-pv-worker232   10Gi       RWO            Delete           Available           yinzhengjie-local-sc            3s
yinzhengjie-pv-worker233   20Gi       RWO            Delete           Available           yinzhengjie-local-sc            3s
[root@master231 yinzhengjie-k8s]# 


	3.手动创建对应节点的目录
[root@worker232 ~]# mkdir -pv /yinzhengjie/data/sc/local01
mkdir: created directory '/yinzhengjie/data/sc'
mkdir: created directory '/yinzhengjie/data/sc/local01'
[root@worker232 ~]# 
[root@worker232 ~]# ll /yinzhengjie/data/sc/local01
total 8
drwxr-xr-x 2 root root 4096 Feb 11 00:36 ./
drwxr-xr-x 3 root root 4096 Feb 11 00:36 ../
[root@worker232 ~]# 



[root@worker233 ~]# mkdir -pv /yinzhengjie/data/sc/local02
mkdir: created directory '/yinzhengjie/data/sc'
mkdir: created directory '/yinzhengjie/data/sc/local02'
[root@worker233 ~]# 
[root@worker233 ~]# ll /yinzhengjie/data/sc/local02
total 8
drwxr-xr-x 2 root root 4096 Feb 11 00:36 ./
drwxr-xr-x 3 root root 4096 Feb 11 00:36 ../
[root@worker233 ~]#

3.创建pvc验证"延迟绑定"

此时我们创建pvc并指定pv，此时按照我们之前的学习应该知道，pvc会关联到对应的pv才对。

但此时会触发WaitForFirstConsumer机制，即"延迟绑定"，什么时候才开始绑定呢？得等到Pod调度完成后才会进行具体的绑定。

因为此时pvc可以关联到worker232和worker233两个节点，如果pvc关联到了worker232，那么将来pod调度到worker233节点，将来在传输数据时就会增加性能的损耗。

而当pod完成调度后，比如调度到worker233节点，那么pvc就会跟着pods调度的节点优先调度到worker233节点的pv，这样做的目的是为了提供pod将来存储的I/O效率。


具体实操验证:
	1.编写资源清单
[root@master231 yinzhengjie-k8s]# cat 03-custom-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: yinzhengjie-pvc-local
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  # pvc无需关联pv，关联sc即可
  storageClassName: yinzhengjie-local-sc
[root@master231 yinzhengjie-k8s]# 


	2.创建pvc
[root@master231 yinzhengjie-k8s]# kubectl apply -f 03-custom-pvc.yaml
persistentvolumeclaim/yinzhengjie-pvc-local created
[root@master231 yinzhengjie-k8s]#


	3.pvc并不会立刻绑定，而是提示"WaitForFirstConsumer"进行了延迟绑定策略
[root@master231 yinzhengjie-k8s]# kubectl get pvc 
NAME                    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS           AGE
yinzhengjie-pvc-local   Pending                                      yinzhengjie-local-sc   4s
[root@master231 yinzhengjie-k8s]# 
[root@master231 yinzhengjie-k8s]# kubectl describe pod 
No resources found in default namespace.
[root@master231 yinzhengjie-k8s]# kubectl describe pvc yinzhengjie-pvc-local 
Name:          yinzhengjie-pvc-local
Namespace:     default
StorageClass:  yinzhengjie-local-sc
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  WaitForFirstConsumer  1s (x2 over 11s)  persistentvolume-controller  waiting for first consumer to be created before binding
[root@master231 yinzhengjie-k8s]#

4.创建pod并观察pvc状态

当pod完成调度后，pvc就会成功调度，也就是我们多次提到的延迟绑定。


具体实操:
	1.编写资源清单
[root@master231 yinzhengjie-k8s]# cat 04-deploy-xiuxian.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xiuxian
spec:
  replicas: 3
  selector:
    matchLabels:
      apps: v1
  template:
    metadata:
      labels:
        apps: v1
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: yinzhengjie-pvc-local
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
        ports: 
        - containerPort: 80
        volumeMounts: 
        - mountPath: "/data"
          name: data
[root@master231 yinzhengjie-k8s]# 


	2.查看pvc状态
[root@master231 yinzhengjie-k8s]# kubectl get pvc 
NAME                    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS           AGE
yinzhengjie-pvc-local   Pending                                      yinzhengjie-local-sc   8m
[root@master231 yinzhengjie-k8s]# 


	3.创建pod后再次观察pvc状态，不难发现，即可完成调度
[root@master231 yinzhengjie-k8s]# kubectl apply -f 04-deploy-xiuxian.yaml 
deployment.apps/xiuxian created
[root@master231 yinzhengjie-k8s]# 
[root@master231 yinzhengjie-k8s]# kubectl get pvc
NAME                    STATUS   VOLUME                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
yinzhengjie-pvc-local   Bound    yinzhengjie-pv-worker233   20Gi       RWO            yinzhengjie-local-sc   8m7s
[root@master231 yinzhengjie-k8s]#  

	
	4.查看pod调度的节点，发现和pvc一样，都调度到worker233节点啦~
[root@master231 yinzhengjie-k8s]# kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE    IP               NODE        NOMINATED NODE   READINESS GATES
xiuxian-7df7678dcf-6ldq4   1/1     Running   0          119s   10.100.140.118   worker233   <none>           <none>
xiuxian-7df7678dcf-mbw2c   1/1     Running   0          119s   10.100.140.121   worker233   <none>           <none>
xiuxian-7df7678dcf-mzttn   1/1     Running   0          119s   10.100.140.70    worker233   <none>           <none>
[root@master231 yinzhengjie-k8s]# 

	5.写入测试数据
[root@master231 yinzhengjie-k8s]# kubectl exec -it xiuxian-7df7678dcf-6ldq4 -- sh
/ # 
/ # cp /etc/hosts  /data/
/ # 
/ # ls -l /data/
total 4
-rw-r--r--    1 root     root           223 Feb 10 16:51 hosts
/ # 
/ # 
[root@master231 yinzhengjie-k8s]# 
[root@master231 yinzhengjie-k8s]# kubectl exec -it xiuxian-7df7678dcf-mzttn -- sh
/ # ls -l /data/
total 4
-rw-r--r--    1 root     root           223 Feb 10 16:51 hosts
/ #

5.删除pod后观察后续pod调度状态

尽管我们多次删除Pod，不难发现pod都会被调度到之前调度的节点。


实战案例:
	1.未删除之前
[root@master231 yinzhengjie-k8s]# kubectl get sc,pvc,pv,po -o wide
NAME                                               PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/yinzhengjie-local-sc   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  24m

NAME                                          STATUS   VOLUME                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE   VOLUMEMODE
persistentvolumeclaim/yinzhengjie-pvc-local   Bound    yinzhengjie-pv-worker233   20Gi       RWO            yinzhengjie-local-sc   12m   Filesystem

NAME                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                           STORAGECLASS           REASON   AGE   VOLUMEMODE
persistentvolume/yinzhengjie-pv-worker232   10Gi       RWO            Delete           Available                                   yinzhengjie-local-sc            19m   Filesystem
persistentvolume/yinzhengjie-pv-worker233   20Gi       RWO            Delete           Bound       default/yinzhengjie-pvc-local   yinzhengjie-local-sc            19m   Filesystem

NAME                           READY   STATUS    RESTARTS   AGE    IP               NODE        NOMINATED NODE   READINESS GATES
pod/xiuxian-7df7678dcf-6ldq4   1/1     Running   0          4m6s   10.100.140.118   worker233   <none>           <none>
pod/xiuxian-7df7678dcf-mbw2c   1/1     Running   0          4m6s   10.100.140.121   worker233   <none>           <none>
pod/xiuxian-7df7678dcf-mzttn   1/1     Running   0          4m6s   10.100.140.70    worker233   <none>           <none>
[root@master231 yinzhengjie-k8s]# 


	2.删除pod
[root@master231 yinzhengjie-k8s]# kubectl delete pods --all
pod "xiuxian-7df7678dcf-6ldq4" deleted
pod "xiuxian-7df7678dcf-mbw2c" deleted
pod "xiuxian-7df7678dcf-mzttn" deleted
[root@master231 yinzhengjie-k8s]# 

	3.发现依旧调度到了worker233节点
[root@master231 yinzhengjie-k8s]# kubectl get sc,pvc,pv,po -o wide
NAME                                               PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/yinzhengjie-local-sc   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  24m

NAME                                          STATUS   VOLUME                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE   VOLUMEMODE
persistentvolumeclaim/yinzhengjie-pvc-local   Bound    yinzhengjie-pv-worker233   20Gi       RWO            yinzhengjie-local-sc   12m   Filesystem

NAME                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                           STORAGECLASS           REASON   AGE   VOLUMEMODE
persistentvolume/yinzhengjie-pv-worker232   10Gi       RWO            Delete           Available                                   yinzhengjie-local-sc            19m   Filesystem
persistentvolume/yinzhengjie-pv-worker233   20Gi       RWO            Delete           Bound       default/yinzhengjie-pvc-local   yinzhengjie-local-sc            19m   Filesystem

NAME                           READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
pod/xiuxian-7df7678dcf-f6gw9   1/1     Running   0          9s    10.100.140.125   worker233   <none>           <none>
pod/xiuxian-7df7678dcf-fzhd4   1/1     Running   0          9s    10.100.140.67    worker233   <none>           <none>
pod/xiuxian-7df7678dcf-kxfwd   1/1     Running   0          9s    10.100.140.77    worker233   <none>           <none>
[root@master231 yinzhengjie-k8s]#

三.local卷个人观点

1.local卷的痛点

个人觉得local卷有一个弊端就是不支持动态创建pv，需要我们运维人员手动编写pv。

其次就是如果k8s集群多个节点有不同的磁盘设备，需要我们运维人员去配置规则来完成对于不同应用的调度任务，未来也是一个比较繁琐的事情。

2.解决local卷的痛点

为了动态解决这些问题，我们需要借助CSI组件来扩展，因为kubelet本身并没有实现。

其中OpenEBS就是一个不错的解决方案。

推荐阅读:
  https://www.cnblogs.com/yinzhengjie/p/18710799
posted @ 2025-02-11 01:35  尹正杰  阅读(180)  评论(0)    收藏  举报