K8S部署Ceph存储集群二

 

 

4.部署ceph集群

4.1 安装cr

cluster(2).yaml

下载集群文件：https://raw.githubusercontent.com/rook/rook/v1.9.9/deploy/examples/cluster.yaml

 

 skipUpgradeChecks: true
  mgr:
    count: 2
  dashboard:
    enabled: true
    ssl: false
  monitoring:
    enabled: true
    rulesNamespace: monitoring
  placement:
    all:
      # nodeAffinity:
      #  requiredDuringSchedulingIgnoredDuringExecution:
      #    nodeSelectorTerms:
      #    - matchExpressions:
      #      - key: ceph
      #        operator: In
      #        values:
      #        - enable
      tolerations:
      - effect: NoSchedule
        key: ceph
        operator: Equal
        value: enable
      - effect: NoSchedule
        key: dedicated
        operator: Equal
        value: bigdata
  storage:
    nodes:
    - deviceFilter: ^vdb #ceph节点空盘设备
      name: k8s-node-1 #节点名称
      resources: {}
    - deviceFilter: ^vdb
      name: k8s-node-2
      resources: {}
    - deviceFilter: ^vdb
      name: k8s-node-3
      resources: {}
    useAllDevices: false
    useAllNodes: false

 

等待集群部署完成，所有osd pod就绪

如果起不来 请使用通配符 原因未知

4.2 安装CephBlockPool cr

 

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph # namespace:cluster
spec:
  failureDomain: host
  replicated:
    size: 3 #副本数
    requireSafeReplicaSize: true
  parameters:
    compression_mode: none
  mirroring:
    enabled: false
    mode: image
  statusCheck:
    mirror:
      disabled: false
      interval: 60s
  # annotations:
  #  key: value

 

4.3 安装StorageClass

 

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  imageFeatures: layering
  imageFormat: "2"
  pool: replicapool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

 

5.部署ceph tools

使用yaml部署，修改下镜像地址加速

rook/toolbox.yaml at v1.8.10 · rook/rook · GitHub

 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-tools
  namespace: rook-ceph # namespace:cluster
  labels:
    app: rook-ceph-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rook-ceph-tools
  template:
    metadata:
      labels:
        app: rook-ceph-tools
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: rook-ceph-tools
          image: rook/ceph:v1.8.10
          command: ["/bin/bash"]
          args: ["-m", "-c", "/usr/local/bin/toolbox.sh"]
          imagePullPolicy: IfNotPresent
          tty: true
          securityContext:
            runAsNonRoot: true
            runAsUser: 2016
            runAsGroup: 2016
          env:
            - name: ROOK_CEPH_USERNAME
              valueFrom:
                secretKeyRef:
                  name: rook-ceph-mon
                  key: ceph-username
            - name: ROOK_CEPH_SECRET
              valueFrom:
                secretKeyRef:
                  name: rook-ceph-mon
                  key: ceph-secret
          volumeMounts:
            - mountPath: /etc/ceph
              name: ceph-config
            - name: mon-endpoint-volume
              mountPath: /etc/rook
      volumes:
        - name: mon-endpoint-volume
          configMap:
            name: rook-ceph-mon-endpoints
            items:
              - key: data
                path: mon-endpoints
        - name: ceph-config
          emptyDir: {}
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5