k8s通过nfs服务pvc动态搭建mongdb复制集群

<1>先为nfs服务创建rbac

Kubectl create -f nfs.rbac.yaml

命名空间(namespace 为defaul）

kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

<2>发布nfs服务

Kubectl create -f nfs_deploy.yaml

因为nfs服务器搭建在master节点上面 所以我把nfs服务通过label标签调度在master节点

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  selector:
     matchLabels:
       app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      nodeSelector:
        node: meiweimin
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 172.17.2.91  # 替换为你的NFS服务器
            - name: NFS_PATH
              value: /var/nfs/public  # 替换为你的NFS文件夹
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.17.2.91  # 替换为你的NFS服务器
            path: /var/nfs/public  # 替换为你的NFS文件夹

 <3>为mongdb创建StorageClass

Kubectl create -f mongo-pv.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: fuseim.pri/ifs
reclaimPolicy: Retain

<3>在创建集群时先为创造权限 在这里我们使用的namespace为default

 Kubectl create -f k8s/pv/nfs/test.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: default-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default

 

<5>创建mongdb复制集群

Kubectl create -f nfs_mongdb.yaml

在这里依然通过label设置了标签  控制pod的调度的范围

因为我service服务是Nodeport 所以如果多个pod被调度在同一个的工作节点的话就会冲突

所以通过affinity属性控制一个节点只能运行一个pod

apiVersion: v1
kind: Service
metadata:
  name: mongodb
spec:
  type: NodePort
  ports:
  - port: 27017
    targetPort: 27017
    nodePort: 27017
  selector:
    k8s-app:  mongodb

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
spec:
  serviceName: mongodb
  replicas: 3
  volumeClaimTemplates:
  - metadata:
      name: pvc
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName:  managed-nfs-storage
      resources:
        requests:
          storage: 400Mi
  
  selector:
    matchLabels:
      k8s-app:  mongodb
  template:
    metadata:
      labels:
        k8s-app: mongodb
        role: mongo
        environment:  test
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: k8s-app
                operator: In
                values:
                - mongodb
            topologyKey: "kubernetes.io/hostname"
      nodeSelector:
        points: mongo
      containers:
      - name: mongo
        image: mongo:3.4
        imagePullPolicy: IfNotPresent
        command:
        - mongod
        - "--replSet"
        - rs0
        - "--bind_ip"
        - 0.0.0.0
        - "--smallfiles"
        - "--noprealloc"
        #- "mongod -f /var/conf.d/mongodb.conf"
        ports:
        - containerPort:  27017
          protocol: TCP
        volumeMounts:
        - name: pvc
          mountPath: /data/db
      - name: mongo-sidecar
        image:  cvallance/mongo-k8s-sidecar
        env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  

 

执行结果

 

 

测试

Kubectl exec -it mongodb-0 -- mongo

rs.status()

 

 

使用go连接mongdb集群