k8s对接ceph存储
前提条件：已经部署好ceph集群
本次实验由于环境有限，ceph集群是部署在k8s的master节点上的
一、创建ceph存储池
在ceph集群的mon节点上执行以下命令：
ceph osd pool create k8s-volumes 64 64
查看下副本数
[root@master ceph]# ceph osd pool get k8s-volumes size size: 3
pg的设置参照以下公式：
Total PGs = ((Total_number_of_OSD * 100) / max_replication_count) / pool_count
结算的结果往上取靠近2的N次方的值。比如总共OSD数量是2，复制份数3，pool数量也是1，那么按上述公式计算出的结果是66.66。取跟它接近的2的N次方是64，那么每个pool分配的PG数量就是64。
二、在k8s的所有节点上安装ceph-common
1、配置国内 yum源地址、ceph源地址
cp -r /etc/yum.repos.d/ /etc/yum-repos-d-bak yum install -y wget rm -rf /etc/yum.repos.d/* wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo yum clean all yum makecache
cat <<EOF > /etc/yum.repos.d/ceph.repo [ceph] name=Ceph packages baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/ enabled=1 gpgcheck=1 priority=1 type=rpm-md gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc [ceph-noarch] name=Ceph noarch packages baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch enabled=1 gpgcheck=1 priority=1 type=rpm-md gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS enabled=0 gpgcheck=1 type=rpm-md gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc priority=1 EOF
2、安装ceph-common
yum -y install ceph-common
3、将ceph的mon节点的配置文件/etc/ceph/ceph.conf 放到所有k8s节点的/etc/ceph目录下
4、将ceph的mon节点的文件 /etc/ceph/ceph.client.admin.keyring 放到所有k8s节点的/etc/ceph目录下
三、以rbac方式对接ceph
由于是用kubeadm部署的k8s集群，kube-controller-manager是以容器方式运行的，里面并没有ceph-common，所以采用 扩展存储卷插件 的方式来实现。
简单说一下，这里提供 rbac 和 no-rbac 两种方式，这里因为我们搭建的 k8s 集群时开启了 rbac 认证的，所以这里采用 rbac 方式来创建该 deployment。
1、下载插件镜像：(本人已经将其上传到阿里云的镜像仓库了)
docker pull registry.cn-hangzhou.aliyuncs.com/boshen-ns/rbd-provisioner:v1.0
2、创建/root/k8s-ceph-rbac/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
3、创建/root/k8s-ceph-rbac/clusterrole.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: rbd-provisioner [root@master k8s-ceph-rbac]# vim clusterrole.yaml [root@master k8s-ceph-rbac]# cat clusterrole.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rbd-provisioner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns","coredns"] verbs: ["list", "get"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get", "create", "delete"]
4、创建/root/k8s-ceph-rbac/clusterrolebinding.yaml
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: rbd-provisioner subjects: - kind: ServiceAccount name: rbd-provisioner namespace: default roleRef: kind: ClusterRole name: rbd-provisioner apiGroup: rbac.authorization.k8s.io
5、创建/root/k8s-ceph-rbac/role.yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbd-provisioner rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]
6、创建/root/k8s-ceph-rbac/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbd-provisioner roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbd-provisioner subjects: - kind: ServiceAccount name: rbd-provisioner namespace: default
7、创建/root/k8s-ceph-rbac/deployment.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: rbd-provisioner spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: rbd-provisioner spec: containers: - name: rbd-provisioner image: registry.cn-hangzhou.aliyuncs.com/boshen-ns/rbd-provisioner:v1.0 imagePullPolicy: Never env: - name: PROVISIONER_NAME value: ceph.com/rbd serviceAccount: rbd-provisioner
8、创建/root/k8s-ceph-rbac/ceph-secret.yaml
apiVersion: v1 kind: Secret metadata: name: ceph-secret type: "kubernetes.io/rbd" data: key: QVFDQmRvbGNxSHlaQmhBQW45WllIbCtVd2JrTnlPV0xseGQ4RUE9PQ==
上面key的值用以下方式查看：
[root@master ~]# grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF}'|base64 QVFDQmRvbGNxSHlaQmhBQW45WllIbCtVd2JrTnlPV0xseGQ4RUE9PQ==
9、创建/root/k8s-ceph-rbac/ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-storage-class provisioner: ceph.com/rbd parameters: #monitors: 192.168.137.10:6789 monitors: ceph-mon-1.default.svc.cluster.local.:6789 adminId: admin adminSecretName: ceph-secret adminSecretNamespace: default pool: k8s-volumes userId: admin userSecretName: ceph-secret fsType: ext4 imageFormat: "2" imageFeatures: layering
注意：上面的monitors，不能直接写ip，这样以后创建pvc会报：missing Ceph monitors。源码中，monitors需要k8s dns解析，我这里使用外部ceph，肯定没有相关解析。所以手动添加解析，如第10步。
10、创建/root/k8s-ceph-rbac/rbd-monitors-dns.yaml
kind: Service apiVersion: v1 metadata: name: ceph-mon-1 spec: type: ExternalName externalName: 192.168.137.10.xip.io
ceph的mon地址为：192.168.137.10:6789
11、执行以下命令将上面1到10步的yaml文件进行执行
kubeclt apply -f k8s-ceph-rbac/
12、进行测试是否可用
1）创建test-pvc.yaml
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: test-pvc spec: accessModes: - ReadWriteOnce storageClassName: ceph-storage-class resources: requests: storage: 1Gi
kubectl apply -f test-pvc.yaml
状态为Bound，说明创建的pvc正常
扩展链接：https://github.com/kubernetes-incubator/external-storage
