cephfs文件系统场景
创建cephfs文件系统：
[cephfsd@ceph-admin ceph]$ cd /etc/ceph
[cephfsd@ceph-admin ceph]$ ceph fs ls
No filesystems enabled
# 创建三个mds
[cephfsd@ceph-admin ceph]$ ceph-deploy mds create ceph-node1 ceph-node2 ceph-node3
# cephfs需要两个pool：
# 1、data pool：存放object
# 2、meta data pool：存放元数据，可设置较高副本级别，也可调整pool的crush_ruleset，使其在ssd上存储，加快客户端响应速度，我这里直接使用默认crush_ruleset
# 创建pool存储池，这里两个pg_num必须保持一致，否则后面无法挂载。
[cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_data 64
pool 'cephfs_data' created
[cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_metadata 64
pool 'cephfs_metadata' created
# 注：
# 计算pg数量：
# 1、集群pg 总数 = （OSD 总数* 100 ）/最大副本数
# 2、每个pool中pg总数=（OSD总数*100）/ 最大副本数 ）/ 池数
# 3、pg数需要是2的指数幂
# 创建fs，这里new后面的cephfs就是文件系统名，可自定义。后面接的cephfs_metadata cephfs_data 就是上面创建的pool。
[cephfsd@ceph-admin ceph]$ ceph fs new cephfs cephfs_metadata cephfs_data
new fs with metadata pool 6 and data pool 5
# 创建成功，查看创建后的fs
[cephfsd@ceph-admin ceph]$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
# 查看mds节点状态，active是活跃的，另1个是处于热备份的状态
[cephfsd@ceph-admin ceph]$ ceph mds stat
cephfs-1/1/1 up {0=ceph-node3=up:active}, 2 up:standby
[cephfsd@ceph-admin ceph]$挂载cephfs文件系统（在任意客户端，我们这里直接在ceph的admin主机上挂载了）
# 创建挂载点
[cephfsd@ceph-admin ceph]$ mkdir /opt
# 去admin主机查看存储秘钥
[cephfsd@ceph-admin ceph]$ cat ceph.client.admin.keyring
[client.admin]
key = AQBIH+ld1okAJhAAmULVJM4zCCVAK/Vdi3Tz5Q==
# 将key的值复制下来，保存到客户端，我们这里保存在了/etc/ceph/admin.secret
[cephfsd@ceph-admin ceph]$ cat admin.secret
AQBIH+ld1okAJhAAmULVJM4zCCVAK/Vdi3Tz5Q==
# 挂载，有两种方式挂载，mount挂载和ceph-fuse挂载
# A.内核驱动挂载Ceph文件系统
[cephfsd@ceph-admin ceph]$ sudo mount.ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret
[cephfsd@ceph-admin ceph]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 484M 0 484M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 26M 470M 6% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 46G 1.9G 44G 4% /
/dev/sda1 xfs 497M 130M 368M 26% /boot
tmpfs tmpfs 100M 0 100M 0% /run/user/0
172.16.143.121:6789:/ ceph 8.5G 0 8.5G 0% /opt
# 注：
# mount时，mon节点有几个写几个，如果配置了hosts解析，可以使用主机名代替IP。
# 还可以使用fuse挂载方式，fuse其实坑挺多的，能不用暂时不用
# 由于kernel挂载方式目前不支持quota的配置，如果需要quota的配置，那么只好使用fuse挂载
# 卸载
[cephfsd@ceph-admin ceph]$ umount /opt
umount: /opt: umount failed: Operation not permitted
[cephfsd@ceph-admin ceph]$ sudo umount /opt
[cephfsd@ceph-admin ceph]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 484M 0 484M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 26M 470M 6% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 46G 1.9G 44G 4% /
/dev/sda1 xfs 497M 130M 368M 26% /boot
tmpfs tmpfs 100M 0 100M 0% /run/user/0
# 设置开机自动挂载/etc/fstab
[cephfsd@ceph-admin ceph]$ sudo vim /etc/fstab
172.16.143.121:6789:/ /opt ceph name=admin,secretfile=/etc/ceph/admin.secret,noatime,_netdev 0 2
# 注意：检查是否启用cephx认证方法，如果值为none为禁用，cephx为启用
[root@node1 ceph]# cat /etc/ceph/ceph.conf | grep auth | grep required
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
# B.用户控件挂载Ceph文件系统
# 安装ceph-fuse
[cephfsd@ceph-admin ceph]$ yum install -y ceph-fuse
# 挂载，挂载需要sudo权限，否则会报错。
[cephfsd@ceph-admin ceph]$ sudo ceph-fuse -m 172.16.143.121:6789 /opt
2019-12-06 08:29:52.137542 7f0e2edb20c0 -1 init, newargv = 0x55c37f3887e0 newargc=9ceph-fuse[50371]: starting ceph client
ceph-fuse[50371]: starting fuse
[cephfsd@ceph-admin ceph]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 484M 0 484M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 26M 470M 6% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 46G 1.9G 44G 4% /
/dev/sda1 xfs 497M 130M 368M 26% /boot
tmpfs tmpfs 100M 0 100M 0% /run/user/0
ceph-fuse fuse.ceph-fuse 8.5G 0 8.5G 0% /opt
# 卸载
[cephfsd@ceph-admin ceph]$ fusermount -u /opt
fusermount: entry for /opt not found in /etc/mtab
[cephfsd@ceph-admin ceph]$ sudo fusermount -u /opt
[cephfsd@ceph-admin ceph]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 484M 0 484M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 26M 470M 6% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 46G 1.9G 44G 4% /
/dev/sda1 xfs 497M 130M 368M 26% /boot
tmpfs tmpfs 100M 0 100M 0% /run/user/0
[cephfsd@ceph-admin ceph]$
# 这里就是用ceph-fuse挂载时没有使用root权限，就挂载不上。
[cephfsd@ceph-admin ceph]$ ceph-fuse -m 172.16.143.121:6789 /opt
2019-12-06 08:29:28.606085 7f35fd5e60c0 -1 asok(0x5627d23481c0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (13) Permission denied
2019-12-06 08:29:28.606957 7f35fd5e60c0 -1 init, newargv = 0x5627d24967e0 newargc=9
ceph-fuse[50325]: starting ceph client
fusermount: user has no write access to mountpoint /opt
ceph-fuse[50325]: fuse failed to start
2019-12-06 08:29:31.044358 7f35fd5e60c0 -1 fuse_mount(mountpoint=/opt) failed.
[cephfsd@ceph-admin ceph]$
[cephfsd@ceph-admin ceph]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 484M 0 484M 0% /dev
tmpfs tmpfs 496M 0 496M 0% /dev/shm
tmpfs tmpfs 496M 26M 470M 6% /run
tmpfs tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 46G 1.9G 44G 4% /
/dev/sda1 xfs 497M 130M 368M 26% /boot
tmpfs tmpfs 100M 0 100M 0% /run/user/0
[cephfsd@ceph-admin ceph]$
# 设置开机自动挂载/etc/fstab
[cephfsd@ceph-admin ceph]$ sudo vim /etc/fstab
id=admin,conf=/etc/ceph/ceph.conf /mnt fuse.ceph defaults 0 0删除cephfs，需要先将mds置为failed。
[cephfsd@ceph-admin ceph]$ ceph fs rm 128
Error EINVAL: all MDS daemons must be inactive before removing filesystem
[cephfsd@ceph-admin ceph]$ ceph mds stat
128-1/1/1 up {0=ceph-node2=up:active(laggy or crashed)}
[cephfsd@ceph-admin ceph]$ ceph mds fail 0
failed mds gid 4147
[cephfsd@ceph-admin ceph]$ ceph mds stat
128-0/1/1 up , 1 failed
[cephfsd@ceph-admin ceph]$ ceph fs rm 128
Error EPERM: this is a DESTRUCTIVE operation and will make data in your filesystem permanently inaccessible. Add --yes-i-really-mean-it if you are sure you wish to continue.
[cephfsd@ceph-admin ceph]$ ceph fs rm 128 --yes-i-really-mean-it
[cephfsd@ceph-admin ceph]$ ceph fs ls
No filesystems enabled
[cephfsd@ceph-admin ceph]$删除pg pool，需要先将使用该pool的cephfs删除。
[cephfsd@ceph-admin ceph]$ ceph osd pool rm cephfs_data2 cephfs_data2 --yes-i-really-mean-it
Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool cephfs_data2. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it.
[cephfsd@ceph-admin ceph]$ ceph osd pool rm cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it
Error EBUSY: pool 'cephfs_data2' is in use by CephFS
# 该pool被cephfs使用了，所以需要先删除该cephfs。
# 因为cephfs_data cephfs_metadata是挂载在128下的，cephfs_data2 cephfs_metadata2是挂载在cephfs下的，所以，删除cephfs
# 接下来，删除cephfs
[cephfsd@ceph-admin ceph]$ ceph fs rm cephfs
Error EPERM: this is a DESTRUCTIVE operation and will make data in your filesystem permanently inaccessible. Add --yes-i-really-mean-it if you are sure you wish to continue.
[cephfsd@ceph-admin ceph]$ ceph fs rm cephfs --yes-i-really-mean-it
[cephfsd@ceph-admin ceph]$ ceph fs ls
name: 128, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[cephfsd@ceph-admin ceph]$
# cephfs删除了，接下来删除pool cephfs_data2
[cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2
Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool cephfs_data2. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it.
[cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
# 这里需要修改配置，增加mon_allow_pool_delete = true
[cephfsd@ceph-admin ceph]$ vim ceph.conf
[cephfsd@ceph-admin ceph]$ cat ceph.conf
[global]
fsid = 6d3fd8ed-d630-48f7-aa8d-ed79da7a69eb
mon_initial_members = ceph-admin
mon_host = 172.16.143.121
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
mon_allow_pool_delete = true
osd_pool_default_size = 3
[mgr]
mgr modules = dashboard
# 然后把配置推送到mon节点。（这里其实不需要了，因为mon节点只有一个，部署在ceph-admin里）
[cephfsd@ceph-admin ceph]$ ceph-deploy --overwrite-conf config push ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephfsd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : push
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f946bf4fa28>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['ceph-node1', 'ceph-node2', 'ceph-node3']
[ceph_deploy.cli][INFO ] func : <function config at 0x7f946c18ac08>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.config][DEBUG ] Pushing config to ceph-node1
[ceph-node1][DEBUG ] connection detected need for sudo
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to ceph-node2
[ceph-node2][DEBUG ] connection detected need for sudo
[ceph-node2][DEBUG ] connected to host: ceph-node2
[ceph-node2][DEBUG ] detect platform information from remote host
[ceph-node2][DEBUG ] detect machine type
[ceph-node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to ceph-node3
[ceph-node3][DEBUG ] connection detected need for sudo
[ceph-node3][DEBUG ] connected to host: ceph-node3
[ceph-node3][DEBUG ] detect platform information from remote host
[ceph-node3][DEBUG ] detect machine type
[ceph-node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
# 依然报这个错，因为没有重启mon，所以这里需要重启一下mon服务。重启服务需要root权限。
[cephfsd@ceph-admin ceph]$ systemctl restart ceph-mon.target
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:
==== AUTHENTICATION COMPLETE ===
[cephfsd@ceph-admin ceph]$
# 接下来就可以删除pool了。
[cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_data2 cephfs_data2 --yes-i-really-really-mean-it
pool 'cephfs_data2' removed
[cephfsd@ceph-admin ceph]$ ceph osd pool delete cephfs_metadata2 cephfs_metadata2 --yes-i-really-really-mean-it
pool 'cephfs_metadata2' removed挂载：
[cephfsd@ceph-admin ceph]$ !mount mount -t ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret mount: only root can use "--options" option [cephfsd@ceph-admin ceph]$ mount.ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret mount error 1 = Operation not permitted # 需要root权限执行mount
注意问题：
1、ceph pg_num设置太大。
ceph使用cephfs
[cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_data 128 pool 'cephfs_data' already exists [cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_metadata 128 Error ERANGE: pg_num 128 size 3 would mean 768 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3) [cephfsd@ceph-admin ceph]$
类似这种，应该就是ceph pg_num设置太大了，设置小一点就好了，如下：
[cephfsd@ceph-admin ceph]$ ceph osd pool create cephfs_metadata 64 pool 'cephfs_metadata' created [cephfsd@ceph-admin ceph]$
2、 pg_num只能增加, 不能缩小.
[cephfsd@ceph-admin ceph]$ ceph osd pool set cephfs_data pg_num 64 Error EEXIST: specified pg_num 64 <= current 12
3、cephfs_data与cephfs_metadata的pg_num值必须保持一致，否则，即使pool创建成功，也无法挂载，就像下面一样：
[cephfsd@ceph-admin ceph]$ sudo mount.ceph 172.16.143.121:6789:/ /opt -o name=admin,secretfile=/etc/ceph/admin.secret ^C^C ^C^C ^C ^CKilled [cephfsd@ceph-admin ceph]$ mount error 5 = Input/output error mount error 110 = Connection timed out
参考：http://www.strugglesquirrel.com/2019/04/23/centos7%E9%83%A8%E7%BD%B2ceph/
