1.添加OSD之前环境查看
1.1 查看集群可用的设备【每个设备想要加入到集群，则其大小不得小于5GB】
[root@ceph141 ~]# ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph141 /dev/sdb hdd 300G Yes 3m ago
ceph141 /dev/sdc hdd 500G Yes 3m ago
ceph141 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 3m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph142 /dev/sdb hdd 300G Yes 3m ago
ceph142 /dev/sdc hdd 500G Yes 3m ago
ceph142 /dev/sdd hdd 1000G Yes 3m ago
ceph142 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 3m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph143 /dev/sdb hdd 300G Yes 17s ago
ceph143 /dev/sdc hdd 500G Yes 17s ago
ceph143 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 17s ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
[root@ceph141 ~]#
温馨提示:
如果一个设备想要加入ceph集群，要求满足2个条件:
- 1.设备未被使用;
- 2.设备的存储大小必须大于5GB;
1.2 查看各节点的空闲设备信息
[root@ceph141 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
...
[root@ceph141 ~]#
[root@ceph142 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1000G 0 disk
...
[root@ceph142 ~]#
[root@ceph143 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
...
[root@ceph143 ~]#
1.3 查看集群中OSD列表
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0 root default
[root@ceph141 ~]#
2.添加OSD设备到集群
2.1 添加ceph141、ceph142、ceph143节点的设备到ceph集群（在141节点操作）
[root@ceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdb
Created osd(s) 0 on host 'ceph141'
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdc
Created osd(s) 1 on host 'ceph141'
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdb
Created osd(s) 2 on host 'ceph142'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdc
Created osd(s) 3 on host 'ceph142'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdb
Created osd(s) 4 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdc
Created osd(s) 5 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdd
Created osd(s) 6 on host 'ceph142'
[root@ceph141 ~]#
温馨提示:
- 1.此步骤会在"/var/lib/ceph/<Ceph_Cluster_ID>/osd.<OSD_ID>/fsid"文件中记录对应ceph的OSD编号对应本地的磁盘设备标识。
- 2.比如查看ceph142节点的硬盘和OSD的对应关系如下:
[root@ceph142 ~]# ll -d /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.*
drwx------ 2 167 167 4096 Aug 21 15:18 /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.2/
drwx------ 2 167 167 4096 Aug 21 15:19 /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.3/
drwx------ 2 167 167 4096 Aug 21 15:22 /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.6/
[root@ceph142 ~]#
[root@ceph142 ~]# cat /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.*/fsid
68ff55fb-358a-4014-ba0e-075adb18c6d9
b9096186-53af-4ca0-b233-01fd913bdaba
d4ccefb2-5812-4ca2-97ca-9642ff4539f2
[root@ceph142 ~]#
[root@ceph142 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
└─ceph--bb7e7dd0--d4e2--4da2--9cfb--da1dcd70222d-osd--block--68ff55fb--358a--4014--ba0e--075adb18c6d9
253:1 0 300G 0 lvm
sdc 8:32 0 500G 0 disk
└─ceph--5b511438--e561--456f--a33e--82bfc9c4abfd-osd--block--b9096186--53af--4ca0--b233--01fd913bdaba
253:2 0 500G 0 lvm
sdd 8:48 0 1000G 0 disk
└─ceph--0d9e77e1--051d--4ba6--8274--cfd85e213ab9-osd--block--d4ccefb2--5812--4ca2--97ca--9642ff4539f2
253:3 0 1000G 0 lvm
...
[root@ceph142 ~]#
2.2 查看集群的osd总容量大小
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 3.32048 root default
-3 0.78130 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
-5 1.75789 host ceph142
2 hdd 0.29300 osd.2 up 1.00000 1.00000
3 hdd 0.48830 osd.3 up 1.00000 1.00000
6 hdd 0.97659 osd.6 up 1.00000 1.00000
-7 0.78130 host ceph143
4 hdd 0.29300 osd.4 up 1.00000 1.00000
5 hdd 0.48830 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
2.3 查看集群的大小
[root@ceph141 ~]# ceph -s
cluster:
id: 3cb12fba-5f6e-11ef-b412-9d303a22b70f
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 19m)
mgr: ceph141.cwgrgj(active, since 3h), standbys: ceph142.ymuzfe
osd: 7 osds: 7 up (since 2m), 7 in (since 3m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 188 MiB used, 3.3 TiB / 3.3 TiB avail # 集群大小共计3.3TB!
pgs: 1 active+clean
[root@ceph141 ~]#
注意，若出现以下报错，可能是由于时间不同步导致
cluster:
id: bcf20c96-b28a-11f1-a4bd-bd2db3a1a43d
health: HEALTH_WARN
Failed to place 2 daemon(s)
clock skew detected on mon.ceph143
解决方案：chrony进行同步时间
1.所有节点安装服务
apt -y install chrony
2.ceph141、ceph142、ceph143修改配置
vim /etc/chrony/chrony.conf
...
#pool ntp.ubuntu.com iburst maxsources 4
#pool 0.ubuntu.pool.ntp.org iburst maxsources 1
#pool 1.ubuntu.pool.ntp.org iburst maxsources 1
#pool 2.ubuntu.pool.ntp.org iburst maxsources 2
pool ntp.aliyun.com iburst maxsources 4
3.所有节点重启服务
systemctl restart chronyd
4.所有节点查看配置是否生效
chronyc activity -v
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
7 sources with unknown address
5.再次查看集群
[root@ceph141 ~]# ceph -s
cluster:
id: 3cb12fba-5f6e-11ef-b412-9d303a22b70f
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 84m)
mgr: ceph141.cwgrgj(active, since 4h), standbys: ceph142.ymuzfe
osd: 7 osds: 7 up (since 67m), 7 in (since 68m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 188 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 1 active+clean
[root@ceph141 ~]#
当然，也可以在dashboard中查看OSD信息
https://ceph141:8443/#/osd
浙公网安备 33010602011771号