Ceph RBD的使用笔记

ceph的内容太多了,单独写一篇文章记录自己的一些RBD的学习笔记,其实简书和其他博客上已经记录的非常全面了,但是因为出处都比较分散,所以还是自己把自己的实验记录一下便于以后学习的查阅。也感谢各位大佬的无私分享。

 

1.RBD池的创建和enable

[cephadmin@ceph-node ~]$ ceph osd pool create rbd 32 32
pool 'rbd' created

[cephadmin@ceph-node ~]$ ceph osd pool application enable rbd rbd
enabled application 'rbd' on pool 'rbd'


[cephadmin@ceph-node ~]$ ceph osd pool ls detail
pool 1 'test-pool' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 43 flags hashpspool max_bytes 214748364800 max_objects 50000 stripe_width 0 application rgw
pool 2 'images' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 41 flags hashpspool stripe_width 0
pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 53 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 56 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 58 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 60 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 70 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 8 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 73 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 9 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 85 flags hashpspool stripe_width 0 application rbd

 

2.快设备的操作

rbd create --size 10240 image01 

[cephadmin@ceph-node ~]$ rbd ls
image01
[cephadmin@ceph-node ~]$ rbd info image01
rbd image 'image01':
    size 10 GiB in 2560 objects
    order 22 (4 MiB objects)
    id: 1218f6b8b4567
    block_name_prefix: rbd_data.1218f6b8b4567
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Tue Aug 15 11:23:05 2023

[cephadmin@ceph-node ~]$ rbd feature disable image01 exclusive-lock, object-map, fast-diff, deep-flatten


[root@ceph-node1 ~]# rbd map image01 
/dev/rbd0

[root@ceph-node1 ~]# rbd showmapped
id pool image   snap device    
0  rbd  image01 -    /dev/rbd0 

[root@ceph-node1 ~]# mkfs.xfs /dev/rbd0
Discarding blocks...Done.
meta-data=/dev/rbd0              isize=512    agcount=16, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


[root@ceph-node1 ~]# mount /dev/rbd0 /mnt

[root@ceph-node1 ~]# cd /mnt
[root@ceph-node1 mnt]# ls
[root@ceph-node1 mnt]# df -TH|grep rbd0
/dev/rbd0               xfs        11G   35M   11G   1% /mnt

我是在另外一个osd节点上操作的,需要把ceph.conf和ceph.client.admin.keyring拷贝到/etc/ceph目录下去。

如果是非ceph节点,需要安装ceph-common的包

 

3. 往块中写入一个文件

[root@ceph-node1 mnt]# dd if=/dev/zero of=/mnt/rbd_test bs=1M count=300
300+0 records in
300+0 records out
314572800 bytes (315 MB) copied, 1.54531 s, 204 MB/s

[root@ceph-node1 mnt]# ll
total 307200
-rw-r--r-- 1 root root 314572800 Aug 15 11:47 rbd_test

[root@ceph-node1 mnt]# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    24 GiB     20 GiB      3.9 GiB         16.22 
POOLS:
    NAME                          ID     USED        %USED     MAX AVAIL     OBJECTS 
    test-pool                     1      124 MiB      1.30       9.2 GiB         104 
    images                        2          0 B         0       9.2 GiB           0 
    .rgw.root                     3      1.1 KiB         0       6.2 GiB           4 
    default.rgw.control           4          0 B         0       6.2 GiB           8 
    default.rgw.meta              5      1.3 KiB         0       6.2 GiB           8 
    default.rgw.log               6          0 B         0       6.2 GiB         175 
    default.rgw.buckets.index     7          0 B         0       6.2 GiB           1 
    default.rgw.buckets.data      8      3.1 KiB         0       6.2 GiB           1 
    rbd                           9      314 MiB      3.21       6.2 GiB          99 

 

发现一个比较奇怪的问题,我的每个osd总空间是8G,但是RBD建立块设备是10G,居然也能成功,应该是按照消耗量来的,消耗多少算多少。另外OSD空间不够支持扩展?

 

4.缩容和扩容

[root@ceph-node1 mnt]# rbd --image image01 resize --size  4096  --allow-shrink 
Resizing image: 100% complete...done.

[root@ceph-node1 mnt]# rbd info image01
rbd image 'image01':
    size 4 GiB in 1024 objects
    order 22 (4 MiB objects)
    id: 1218f6b8b4567
    block_name_prefix: rbd_data.1218f6b8b4567
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 15 11:23:05 2023

[root@ceph-node1 mnt]# df -Th
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  908M     0  908M   0% /dev
tmpfs                   tmpfs     919M     0  919M   0% /dev/shm
tmpfs                   tmpfs     919M  8.6M  911M   1% /run
tmpfs                   tmpfs     919M     0  919M   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs       8.0G  1.8G  6.2G  23% /
/dev/sda1               xfs      1014M  181M  834M  18% /boot
tmpfs                   tmpfs     919M   28K  919M   1% /var/lib/ceph/osd/ceph-1
tmpfs                   tmpfs     184M     0  184M   0% /run/user/0
/dev/rbd0               xfs        10G  333M  9.7G   4% /mnt

df来看并没有什么变化,查阅了一些材料,基本上xfs不支持缩容

[root@ceph-node1 mnt]# rbd --image image01 resize --size 20480
Resizing image: 100% complete...done.

[root@ceph-node1 mnt]# rbd info image01
rbd image 'image01':
    size 20 GiB in 5120 objects
    order 22 (4 MiB objects)
    id: 1218f6b8b4567
    block_name_prefix: rbd_data.1218f6b8b4567
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 15 11:23:05 2023

[root@ceph-node1 mnt]# xfs_growfs /mnt
meta-data=/dev/rbd0              isize=512    agcount=16, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Invalid argument
data blocks changed from 2621440 to 5242880


[root@ceph-node1 mnt]# df -Th
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  908M     0  908M   0% /dev
tmpfs                   tmpfs     919M     0  919M   0% /dev/shm
tmpfs                   tmpfs     919M  8.6M  911M   1% /run
tmpfs                   tmpfs     919M     0  919M   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs       8.0G  1.9G  6.2G  23% /
/dev/sda1               xfs      1014M  181M  834M  18% /boot
tmpfs                   tmpfs     919M   28K  919M   1% /var/lib/ceph/osd/ceph-1
tmpfs                   tmpfs     184M     0  184M   0% /run/user/0
/dev/rbd0               xfs        20G  334M   20G   2% /mnt

 

 问题解决

中间虚机出问题突然挂死,然后重启启动以后发现无法mount上了。

主要报错信息

mount: wrong fs type, bad option, bad superblock on /dev/rbd0, missing codepage or helper program, or other error
......


#dmesg一下

[  230.385386] rbd: loaded (major 252)
[  230.390674] libceph: mon0 192.168.56.110:6789 session established
[  230.391311] libceph: client84118 fsid 90933ecf-4a54-475b-8222-b2c57868512a
[  230.471613] rbd: rbd0: capacity 21474836480 features 0x1
[  259.268165] XFS (rbd0): Mounting V5 Filesystem
[  259.268885] XFS (rbd0): totally zeroed log
[  259.269379] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
[  259.269382] XFS (rbd0): log mount/recovery failed: error -22
[  259.269400] XFS (rbd0): log mount failed
[  295.912616] XFS (rbd0): Mounting V5 Filesystem
[  295.913160] XFS (rbd0): totally zeroed log
[  295.913576] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
[  295.913578] XFS (rbd0): log mount/recovery failed: error -22
[  295.913597] XFS (rbd0): log mount failed
[  504.223454] XFS (rbd0): Mounting V5 Filesystem
[  504.225363] XFS (rbd0): totally zeroed log
[  504.225880] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
[  504.225883] XFS (rbd0): log mount/recovery failed: error -22
[  504.225904] XFS (rbd0): log mount failed
[  519.475527] XFS (rbd0): Mounting V5 Filesystem
[  519.475956] XFS (rbd0): totally zeroed log
[  519.476219] XFS (rbd0): Corruption warning: Metadata has LSN (1:512) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
[  519.476235] XFS (rbd0): log mount/recovery failed: error -22
[  519.476256] XFS (rbd0): log mount failed

[root@ceph-node1 /]# xfs_repair /dev/rbd0
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
totally zeroed log
        - scan filesystem freespace and inode maps...
bad magic number
bad magic number
bad magic number

。。。。

[root@ceph-node1 /]# mount /dev/rbd0 /mnt
[root@ceph-node1 /]# cd /mnt
[root@ceph-node1 mnt]# ls
rbd_test

 

5.查看有没有客户端已经mount了这个块设备

[cephadmin@ceph-node ~]$ rbd info image01
rbd image 'image01':
    size 20 GiB in 5120 objects
    order 22 (4 MiB objects)
    id: 1218f6b8b4567
    block_name_prefix: rbd_data.1218f6b8b4567
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 15 11:23:05 2023

 [cephadmin@ceph-node ~]$ rados -p rbd listwatchers rbd_header.1218f6b8b4567
watcher=192.168.56.111:0/2196229061 client.84118 cookie=18446462598732840961

 

 

 

参考链接

https://www.jianshu.com/p/712e58d36a77

https://www.cnblogs.com/zhrx/p/16098075.html

 https://zhuanlan.zhihu.com/p/343376229

 

posted @ 2023-08-15 12:12  ericnie  阅读(129)  评论(0编辑  收藏  举报