k8s使用ceph的rbd作后端存储
k8s使用rbd作后端存储
k8s里的存储方式主要有三种。分别是volume、persistent volumes和dynamic volume provisioning。
- volume: 就是直接挂载在pod上的组件，k8s中所有的其他存储组件都是通过volume来跟pod直接联系的。volume有个type属性，type决定了挂载的存储是什么，常见的比如：emptyDir，hostPath，nfs，rbd，以及下文要说的persistentVolumeClaim等。跟docker里面的volume概念不同的是，docker里的volume的生命周期是跟docker紧紧绑在一起的。这里根据type的不同，生命周期也不同，比如emptyDir类型的就是跟docker一样，pod挂掉，对应的volume也就消失了，而其他类型的都是永久存储。详细介绍可以参考Volumes
- Persistent Volumes：顾名思义，这个组件就是用来支持永久存储的，Persistent Volumes组件会抽象后端存储的提供者（也就是上文中volume中的type）和消费者（即具体哪个pod使用）。该组件提供了PersistentVolume和PersistentVolumeClaim两个概念来抽象上述两者。一个PersistentVolume（简称PV）就是后端存储提供的一块存储空间，具体到ceph rbd中就是一个image，一个PersistentVolumeClaim（简称PVC）可以看做是用户对PV的请求，PVC会跟某个PV绑定，然后某个具体pod会在volume 中挂载PVC,就挂载了对应的PV。关于更多详细信息比如PV,PVC的生命周期，dockerfile 格式等信息参考Persistent Volumes
- Dynamic Volume Provisioning: 动态volume发现，比如上面的Persistent Volumes,我们必须先要创建一个存储块，比如一个ceph中的image，然后将该image绑定PV，才能使用。这种静态的绑定模式太僵硬，每次申请存储都要向存储提供者索要一份存储快。Dynamic Volume Provisioning就是解决这个问题的。它引入了StorageClass这个概念，StorageClass抽象了存储提供者，只需在PVC中指定StorageClass，然后说明要多大的存储就可以了，存储提供者会根据需求动态创建所需存储快。甚至于，我们可以指定一个默认StorageClass，这样，只需创建PVC就可以了。
PV的访问模式
- ReadWriteOnce：可读可写，但只支持被单个node挂载。
- ReadOnlyMany：只读权限，可以以只读的方式被多个node挂载。
- ReadWriteMany：可读可写，这种存储可以以读写的方式被多个node共享。
PV回收策略
- Retain – 手动重新使用
- Recycle – 基本的删除操作 (“rm -rf /thevolume/*”)
- Delete – 关联的后端存储卷一起删除，后端存储例如AWS EBS, GCE PD或OpenStack Cinder
在CLI下，访问方式被简写为：
RWO – ReadWriteOnce
ROX – ReadOnlyMany
RWX – ReadWriteMany
在当前的定义中，这三种方式都是针对节点级别的，也就是说，对于一个Persistent Volume, 如果是RWO, 那么只能被挂载在某一个Kubernetes的工作节点（以下简称节点）上，当再次尝试在其他节点挂载的时候，系统会报Multi-Attach的错误（当然，在只有一台可调度节点的情况，即使RWO也是能够被多个Pod同时使用的，但除了开发测试，有谁会这么用呢？）； 如果是RMX, 那么可以同时在多个节点上挂载并被不同的Pod使用。
官方支持如下：
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
所以，我们这里是无法在多个node节点使用rbd的。
折中解决办法：
方法一：使用 NodeSelector标签，限制pod调度到指定的Node上。（如此便是单机）
方法二：不使用k8s，直接将rbd挂载到单台linux机器上。（也是单机）
开始使用rbd
每个k8s node都需要安装ceph客户端ceph-common，才能正常使用ceph
[root@k8s-node1 yum.repos.d]# cd /etc/yum.repos.d [root@k8s-node1 yum.repos.d]# pwd /etc/yum.repos.d # 这里ceph源最好和ceph存储系统里使用的保持一致，保证ceph客户端版本一致 [root@k8s-node1 yum.repos.d]# vim ceph.repo [Ceph] name=Ceph packages for $basearch baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/$basearch enabled=1 gpgcheck=0 type=rpm-md gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc priority=1 [Ceph-noarch] name=Ceph noarch packages baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch enabled=1 gpgcheck=0 type=rpm-md gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS enabled=1 gpgcheck=0 type=rpm-md gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc priority=1 # 安装ceph-common [root@k8s-node1 yum.repos.d]# yum -y install ceph-common
注：依赖包报错
# 如果yum安装ceph-common的时候，报这个错，则先安装epel-release
[root@k8s-node1 yum.repos.d]# yum -y install ceph-common
...
---> Package python-six.noarch 0:1.9.0-2.el7 will be installed
--> Finished Dependency Resolution
Error: Package: 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph)
Requires: libleveldb.so.1()(64bit)
Error: Package: 2:librados2-12.2.12-0.el7.x86_64 (Ceph)
Requires: liblttng-ust.so.0()(64bit)
Error: Package: 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph)
Requires: libbabeltrace-ctf.so.1()(64bit)
Error: Package: 2:ceph-common-12.2.12-0.el7.x86_64 (Ceph)
Requires: libbabeltrace.so.1()(64bit)
Error: Package: 2:librbd1-12.2.12-0.el7.x86_64 (Ceph)
Requires: liblttng-ust.so.0()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
# 安装epel-release
[root@k8s-node1 yum.repos.d]# yum install epel-release
# 此时，再次安装ceph-common即可安装上
[root@k8s-node1 yum.repos.d]# yum -y install ceph-common
[root@k8s-node1 yum.repos.d]# ceph --version
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)每个k8s-node都必须要安装ceph客户端，否则无法挂载
k8s-master上也安装
[root@k8s-master yum.repos.d]# cd /etc/yum.repos.d [root@k8s-master yum.repos.d]# yum -y install ceph-commo [root@k8s-master yum.repos.d]# ceph --version ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
ceph配置（在ceph-admin节点设置）
# 创建存储池rbd_data，（我们测试环境之前创建过，所以略过）
[cephfsd@ceph-admin ceph]$ ceph osd pool create rbd_data 64 64
# 创建rbd镜像 （因为我们之前配置里指定了format为1，所以这里不需要显示指定）
[cephfsd@ceph-admin ceph]$ rbd create rbd_data/filecenter_image --size=10G
# 映射该rbd镜像
[cephfsd@ceph-admin ceph]$ sudo rbd map rbd_data/filecenter_image
/dev/rbd4
# 查看镜像信息
[cephfsd@ceph-admin ceph]$ rbd info rbd_data/filecenter_image
rbd image 'filecenter_image':
size 10GiB in 2560 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.376b6b8b4567
format: 2
features: layering
flags:
create_timestamp: Sat Dec 7 17:37:41 2019
[cephfsd@ceph-admin ceph]$
# 创建secret:
# 在创建pv前,由于ceph是开启了cephx认证的,于是首先需要创建secret资源,k8s的secret资源采用的是base64加密
# 在ceph monitor上提取key:
# 生成加密的key
[cephfsd@ceph-admin ceph]$ cat ceph.client.admin.keyring
[client.admin]
key = AQBIH+ld1okAJhAAmULVJM4zCCVAK/Vdi3Tz5Q==
[cephfsd@ceph-admin ceph]$ ceph auth get-key client.admin | base64
QVFCSUgrbGQxb2tBSmhBQW1VTFZKTTR6Q0NWQUsvVmRpM1R6NVE9PQ==k8s-master上创建ceph的secret
# 创建一个目录，存放yaml文件，自己可以随意设置 [root@k8s-master yum.repos.d]# mkdir /root/k8s/nmp/k1/ceph [root@k8s-master yum.repos.d]# cd /root/k8s/nmp/k1/ceph/ [root@k8s-master ceph]# pwd /root/k8s/nmp/k1/ceph # 创建secret [root@k8s-master ceph]# vim ceph-secret.yaml [root@k8s-master ceph]# cat ceph-secret.yaml apiVersion: v1 kind: Secret metadata: name: ceph-secret type: "kubernetes.io/rbd" data: key: QVFDTTlXOWFOMk9IR3hBQXZyUjFjdGJDSFpoZUtmckY0N2tZOUE9PQ== [root@k8s-master ceph]# kubectl create -f ceph-secret.yaml secret "ceph-secret" created [root@k8s-master ceph]# kubectl get secret NAME TYPE DATA AGE ceph-secret kubernetes.io/rbd 1 7s [root@k8s-master ceph]#
创建PV
[root@k8s-master ceph]# vim filecenter-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: filecenter-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- 172.16.143.121:6789
pool: rbd_data
image: filecenter_image
user: admin
secretRef:
name: ceph-secret
fsType: xfs
readOnly: false
persistentVolumeReclaimPolicy: Recycle
[root@k8s-master ceph]# kubectl create -f filecenter-pv.yaml
persistentvolume "filecenter-pv" created
[root@k8s-master ceph]# kubectl get pv -o wide
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
filecenter-pv 1Gi RWO Recycle Available 18m创建PVC
[root@k8s-master ceph]# vim filecenter-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: filecenter-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
[root@k8s-master ceph]# kubectl create -f filecenter-pvc.yaml
persistentvolumeclaim "filecenter-pvc" created
[root@k8s-master ceph]# kubectl get pvc -o wide
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
filecenter-pvc Bound filecenter-pv 1Gi RWO 6s
[root@k8s-master ceph]#创建deployment挂载PVC
这里修改之前已经在使用的php-filecenter.yaml
[root@k8s-master ceph]# vim ../php/file-center/php-filecenter-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: php-filecenter-deployment
spec:
replicas: 1
selector:
matchLabels:
app: php-filecenter
template:
metadata:
labels:
app: php-filecenter
spec:
containers:
- name: php-filecenter
image: 172.16.143.107:5000/php-fpm:v2019120205
volumeMounts:
- mountPath: "/mnt"
name: filedata
volumes:
- name: filedata
persistentVolumeClaim:
claimName: filecenter-pvc
[root@k8s-master ceph]# kubectl apply -f ../php/file-center/php-filecenter-deployment.yaml
deployment "php-filecenter-deployment" configured
[root@k8s-master ceph]#报错：pvc没有创建成功，导致pod创建失败
# pvc并没有绑定成功，导致pod创建失败
[root@k8s-master ceph]# kubectl exec -it php-filecenter-deployment-3316474311-g1jmg bash
Error from server (BadRequest): pod php-filecenter-deployment-3316474311-g1jmg does not have a host assigned
[root@k8s-master ceph]# kubectl describe pod php-filecenter-deployment-3316474311-g1jmg
Name: php-filecenter-deployment-3316474311-g1jmg
Namespace: default
Node: /
Labels: app=php-filecenter
pod-template-hash=3316474311
Status: Pending
IP:
Controllers: ReplicaSet/php-filecenter-deployment-3316474311
Containers:
php-filecenter:
Image: 172.16.143.107:5000/php-fpm:v2019120205
Port:
Volume Mounts:
/mnt from filedata (rw)
Environment Variables: <none>
Conditions:
Type Status
PodScheduled False
Volumes:
filedata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: filecenter-pvc
ReadOnly: false
QoS Class: BestEffort
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
8m 1m 29 {default-scheduler } Warning FailedScheduling [SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "filecenter-pvc", which is unexpected., SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "filecenter-pvc", which is unexpected.]
# 查看pv，pv是Available
[root@k8s-master ceph]# kubectl get pv -o wide
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
filecenter-pv 1Gi RWO Recycle Available 39m
# 查看pvc，发现pvc是pending状态，有问题
[root@k8s-master ceph]# kubectl get pvc -o wide
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
filecenter-pvc Pending 35m
[root@k8s-master ceph]# kubectl get pod php-filecenter-deployment-3316474311-g1jmg
NAME READY STATUS RESTARTS AGE
php-filecenter-deployment-3316474311-g1jmg 0/1 Pending 0 9m
[root@k8s-master ceph]#
[root@k8s-master ceph]# kubectl describe pv filecenter-pv
Name: filecenter-pv
Labels: <none>
StorageClass:
Status: Available
Claim:
Reclaim Policy: Recycle
Access Modes: RWO
Capacity: 1Gi
Message:
Source:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [172.16.143.121:6789]
RBDImage: filecenter_image
FSType: xfs
RBDPool: rbd_data
RadosUser: admin
Keyring: /etc/ceph/keyring
SecretRef: &{ceph-secret}
ReadOnly: false
No events.
[root@k8s-master ceph]# kubectl describe pvc filecenter-pvc
Name: filecenter-pvc
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: <none>
Capacity:
Access Modes:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
48m 5s 196 {persistentvolume-controller } Normal FailedBinding no persistent volumes available for this claim and no storage class is set
[root@k8s-master ceph]#
# 这种问题就表示pv和pvc没有绑定成功，如果没有指定match labels等规则，那要么就是容量大小不匹配，要么就是access modes不匹配
# 这里检查，发现是pv定义的1G，而pvc定义的是10G，容量大小不匹配，所以pvc没有绑定成功。
# 修改pvc的配置
[root@k8s-master ceph]# vim filecenter-pvc.yaml
# 再次重新应用配置
[root@k8s-master ceph]# kubectl apply -f filecenter-pvc.yaml
The PersistentVolumeClaim "filecenter-pvc" is invalid: spec: Forbidden: field is immutable after creation
# 删除pvc，重新创建
[root@k8s-master ceph]# kubectl delete -f filecenter-pvc.yaml
persistentvolumeclaim "filecenter-pvc" deleted
# 再次检查配置是否一致
[root@k8s-master ceph]# cat filecenter-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: filecenter-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
[root@k8s-master ceph]# cat filecenter-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: filecenter-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- 172.16.143.121:6789
pool: rbd_data
image: filecenter_image
user: admin
secretRef:
name: ceph-secret
fsType: xfs
readOnly: false
persistentVolumeReclaimPolicy: Recycle
# 重新创建pvc
[root@k8s-master ceph]# kubectl create -f filecenter-pvc.yaml
persistentvolumeclaim "filecenter-pvc" created
[root@k8s-master ceph]# kubectl get pvc -o wide
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
filecenter-pvc Bound filecenter-pv 1Gi RWO 6s
[root@k8s-master ceph]#
# 可以看到pvc是Bound状态，绑定成功。没有rbd map，导致pod绑定rbd失败
[root@k8s-master ceph]# kubectl apply -f ../php/file-center/php-filecenter-deployment.yaml
service "php-filecenter-service" configured
deployment "php-filecenter-deployment" configured
[root@k8s-master ceph]# kubectl get pod php-filecenter-deployment-3316474311-g1jmg
NAME READY STATUS RESTARTS AGE
php-filecenter-deployment-3316474311-g1jmg 0/1 ContainerCreating 0 41m
[root@k8s-master ceph]# kubectl logs -f php-filecenter-deployment-3316474311-g1jmg
Error from server (BadRequest): container "php-filecenter" in pod "php-filecenter-deployment-3316474311-g1jmg" is waiting to start: ContainerCreating
[root@k8s-master ceph]# kubectl describe pod php-filecenter-deployment-3316474311-g1jmg
Name: php-filecenter-deployment-3316474311-g1jmg
Namespace: default
Node: k8s-node1/172.16.143.108
Start Time: Sat, 07 Dec 2019 18:52:30 +0800
Labels: app=php-filecenter
pod-template-hash=3316474311
Status: Pending
IP:
Controllers: ReplicaSet/php-filecenter-deployment-3316474311
Containers:
php-filecenter:
Container ID:
Image: 172.16.143.107:5000/php-fpm:v2019120205
Image ID:
Port:
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Volume Mounts:
/mnt from filedata (rw)
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
filedata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: filecenter-pvc
ReadOnly: false
QoS Class: BestEffort
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
41m 4m 133 {default-scheduler } Warning FailedScheduling [SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "filecenter-pvc", which is unexpected., SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "filecenter-pvc", which is unexpected.]
3m 3m 1 {default-scheduler } Normal Scheduled Successfully assigned php-filecenter-deployment-3316474311-g1jmg to k8s-node1
1m 1m 1 {kubelet k8s-node1} Warning FailedMount Unable to mount volumes for pod "php-filecenter-deployment-3316474311-g1jmg_default(5d5d48d9-18da-11ea-8c36-000c29fc3a73)": timeout expired waiting for volumes to attach/mount for pod "default"/"php-filecenter-deployment-3316474311-g1jmg". list of unattached/unmounted volumes=[filedata]
1m 1m 1 {kubelet k8s-node1} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"php-filecenter-deployment-3316474311-g1jmg". list of unattached/unmounted volumes=[filedata]
1m 1m 1 {kubelet k8s-node1} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/5d5d48d9-18da-11ea-8c36-000c29fc3a73-filecenter-pv" (spec.Name: "filecenter-pv") pod "5d5d48d9-18da-11ea-8c36-000c29fc3a73" (UID: "5d5d48d9-18da-11ea-8c36-000c29fc3a73") with: rbd: map failed exit status 1 2019-12-07 18:54:44.268690 7f2f99d00d40 -1 did not load config file, using default settings.
2019-12-07 18:54:44.271606 7f2f99d00d40 -1 Errors while parsing config file!
2019-12-07 18:54:44.271610 7f2f99d00d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-12-07 18:54:44.271610 7f2f99d00d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-12-07 18:54:44.271610 7f2f99d00d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2019-12-07 18:54:44.272599 7f2f99d00d40 -1 Errors while parsing config file!
2019-12-07 18:54:44.272603 7f2f99d00d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-12-07 18:54:44.272603 7f2f99d00d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-12-07 18:54:44.272604 7f2f99d00d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2019-12-07 18:54:44.291155 7f2f99d00d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: sysfs write failed
2019-12-07 18:54:44.297026 7f2f99d00d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-07 18:54:44.298627 7f2f99d00d40 0 librados: client.admin authentication error (1) Operation not permitted
rbd: couldn't connect to the cluster!
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
# 上面说没有ceph.conf和ceph.client.admin.keyring，我们从ceph-admin节点拷贝过来
[root@k8s-master ceph]# rz
[root@k8s-master ceph]# ls
ceph.client.admin.keyring ceph.conf rbdmap
# k8s所有节点都要拷贝，node节点也拷贝一下
[root@k8s-node1 ceph]# rz
[root@k8s-node1 ceph]# ls
ceph.client.admin.keyring ceph.conf rbdmap
# 删除pod，让它重新挂载一下
[root@k8s-master ceph]# kubectl delete pod php-filecenter-deployment-3316474311-g1jmg
pod "php-filecenter-deployment-3316474311-g1jmg" deleted
[root@k8s-master ceph]# kubectl describe pod php-filecenter-deployment-3316474311-jr48g
Name: php-filecenter-deployment-3316474311-jr48g
Namespace: default
Node: k8s-master/172.16.143.107
Start Time: Mon, 09 Dec 2019 10:01:29 +0800
Labels: app=php-filecenter
pod-template-hash=3316474311
Status: Pending
IP:
Controllers: ReplicaSet/php-filecenter-deployment-3316474311
Containers:
php-filecenter:
Container ID:
Image: 172.16.143.107:5000/php-fpm:v2019120205
Image ID:
Port:
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Volume Mounts:
/mnt from filedata (rw)
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
filedata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: filecenter-pvc
ReadOnly: false
QoS Class: BestEffort
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
10s 10s 1 {default-scheduler } Normal Scheduled Successfully assigned php-filecenter-deployment-3316474311-jr48g to k8s-master
8s 8s 1 {kubelet k8s-master} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/d05aa080-1a27-11ea-8c36-000c29fc3a73-filecenter-pv" (spec.Name: "filecenter-pv") pod "d05aa080-1a27-11ea-8c36-000c29fc3a73" (UID: "d05aa080-1a27-11ea-8c36-000c29fc3a73") with: rbd: map failed exit status 1 rbd: sysfs write failed
2019-12-09 10:01:30.443054 7f96b803fd40 0 librados: client.admin authentication error (1) Operation not permitted
rbd: couldn't connect to the cluster!
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
6s 6s 1 {kubelet k8s-master} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/d05aa080-1a27-11ea-8c36-000c29fc3a73-filecenter-pv" (spec.Name: "filecenter-pv") pod "d05aa080-1a27-11ea-8c36-000c29fc3a73" (UID: "d05aa080-1a27-11ea-8c36-000c29fc3a73") with: rbd: map failed exit status 1 rbd: sysfs write failed
2019-12-09 10:01:32.022514 7fb376cb0d40 0 librados: client.admin authentication error (1) Operation not permitted
rbd: couldn't connect to the cluster!
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
4s 4s 1 {kubelet k8s-master} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/d05aa080-1a27-11ea-8c36-000c29fc3a73-filecenter-pv" (spec.Name: "filecenter-pv") pod "d05aa080-1a27-11ea-8c36-000c29fc3a73" (UID: "d05aa080-1a27-11ea-8c36-000c29fc3a73") with: rbd: map failed exit status 1 rbd: sysfs write failed
2019-12-09 10:01:34.197942 7f0282d5fd40 0 librados: client.admin authentication error (1) Operation not permitted
rbd: couldn't connect to the cluster!
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
1s 1s 1 {kubelet k8s-master} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/d05aa080-1a27-11ea-8c36-000c29fc3a73-filecenter-pv" (spec.Name: "filecenter-pv") pod "d05aa080-1a27-11ea-8c36-000c29fc3a73" (UID: "d05aa080-1a27-11ea-8c36-000c29fc3a73") with: rbd: map failed exit status 1 rbd: sysfs write failed
2019-12-09 10:01:37.602709 7f18facc1d40 0 librados: client.admin authentication error (1) Operation not permitted
rbd: couldn't connect to the cluster!
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
[root@k8s-master ceph]#
