ceph部署

后续只使用rbd功能，所以下面的部署过程，只对deploy-ceph/mon/osd/mgr/mds进行记录

 

操作系统 centos7 

内核版本 3.10.0-957.10.1.el7.x86_64

ceph version 12.2.12 luminous 

ceph-deploy-2.0.1-0

 

节点信息

ceph-node-005085  10.10.5.85   deploy-ceph

ceph-node-005084  10.10.5.84   mon/osd/mgr/mds

ceph-node-005070  10.10.5.70  mon/osd

ceph-node-005079  10.10.5.79  mon/osd/mgr

 

添加阿里云yum源

cat /etc/yum.repos.d/ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=https://mirrors.aliyun.com/ceph/rpm-luminous/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc

[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc

 

1.1 安装 ceph工具

在ceph-node-005085节点上安装：yum -y install ceph-deploy

所有节点安装ceph rpm 包，我的操作系统直接yum install ceph安装失败，所以使用下面命令手动安装的rpm

sudo rpm -e mlnx-ofa_kernel-4.4-OFED.4.4.2.0.7.1.gee7aa0e.rhel7u5.x86_64 kmod-mlnx-ofa_kernel-4.4-OFED.4.4.2.0.7.1.gee7aa0e.rhel7u5.x86_64

sudo rpm -ivh /tmp/rpm/libibverbs-17.2-3.el7.x86_64.rpm

sudo rpm -ivh /tmp/rpm/rdma-core-17.2-3.el7.x86_64.rpm

sudo yum -y install ceph ceph-radosgw

 

1.2 安装时间同步工具

yum -y install chrony

配置文件：/etc/chrony.conf

如果是内网的设备，将配置文件内的时间服务器修改正确

1.3 在各节点添加ceph-admin用户，并创建信任关系，ceph-node-005085可以无密码登录各节点

略

2.1 创建集群并准备配置文件

[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy new ceph-node-005084 ceph-node-005070 ceph-node-005079

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy new ceph-node-005084 ceph-node-005070 ceph-node-005079
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] func : <function new at 0x7f7d61856320>
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f7d61879b48>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['ceph-node-005084', 'ceph-node-005070', 'ceph-node-005079']
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005085
[ceph-node-005084][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-node-005084
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph-node-005084][INFO ] Running command: sudo /usr/sbin/ip link show
[ceph-node-005084][INFO ] Running command: sudo /usr/sbin/ip addr show
[ceph-node-005084][DEBUG ] IP addresses found: [u'10.254.189.130', u'10.254.229.192', u'10.254.48.134', u'10.254.0.2', u'10.10.5.84', u'10.254.0.1', u'10.254.153.180', u'172.10.0.128', u'10.254.150.54', u'172.10.0.129', u'10.254.147.53', u'10.254.177.166', u'10.254.223.32', u'10.254.80.120', u'10.254.179.132', u'10.254.236.140']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node-005084
[ceph_deploy.new][DEBUG ] Monitor ceph-node-005084 at 10.10.5.84
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph-node-005070][DEBUG ] connected to host: ceph-node-005085
[ceph-node-005070][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-node-005070
[ceph-node-005070][DEBUG ] connection detected need for sudo
[ceph-node-005070][DEBUG ] connected to host: ceph-node-005070
[ceph-node-005070][DEBUG ] detect platform information from remote host
[ceph-node-005070][DEBUG ] detect machine type
[ceph-node-005070][DEBUG ] find the location of an executable
[ceph-node-005070][INFO ] Running command: sudo /usr/sbin/ip link show
[ceph-node-005070][INFO ] Running command: sudo /usr/sbin/ip addr show
[ceph-node-005070][DEBUG ] IP addresses found: [u'10.254.189.130', u'10.254.229.192', u'172.10.11.129', u'10.254.48.134', u'10.10.5.70', u'10.254.0.2', u'10.254.0.1', u'10.254.153.180', u'10.254.150.54', u'10.254.80.120', u'10.254.147.53', u'10.254.177.166', u'10.254.223.32', u'10.254.179.132', u'10.254.236.140', u'172.10.11.128']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node-005070
[ceph_deploy.new][DEBUG ] Monitor ceph-node-005070 at 10.10.5.70
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph-node-005079][DEBUG ] connected to host: ceph-node-005085
[ceph-node-005079][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-node-005079
[ceph-node-005079][DEBUG ] connection detected need for sudo
[ceph-node-005079][DEBUG ] connected to host: ceph-node-005079
[ceph-node-005079][DEBUG ] detect platform information from remote host
[ceph-node-005079][DEBUG ] detect machine type
[ceph-node-005079][DEBUG ] find the location of an executable
[ceph-node-005079][INFO ] Running command: sudo /usr/sbin/ip link show
[ceph-node-005079][INFO ] Running command: sudo /usr/sbin/ip addr show
[ceph-node-005079][DEBUG ] IP addresses found: [u'10.254.189.130', u'10.254.229.192', u'10.254.48.134', u'172.10.13.128', u'10.254.0.2', u'10.10.5.79', u'10.254.0.1', u'172.10.13.129', u'10.254.153.180', u'10.254.150.54', u'10.254.80.120', u'10.254.147.53', u'10.254.177.166', u'10.254.223.32', u'10.254.179.132', u'10.254.236.140']
[ceph_deploy.new][DEBUG ] Resolving host ceph-node-005079
[ceph_deploy.new][DEBUG ] Monitor ceph-node-005079 at 10.10.5.79
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-node-005084', 'ceph-node-005070', 'ceph-node-005079']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.10.5.84', '10.10.5.70', '10.10.5.79']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph-admin@ceph-node-005085 ceph-cluster]$ ls
ceph.bootstrap-mds.keyring ceph.bootstrap-mgr.keyring ceph.bootstrap-osd.keyring ceph.bootstrap-rgw.keyring ceph.client.admin.keyring ceph-cluster ceph.conf ceph-deploy-ceph.log ceph.mon.keyring iplist

 

执行完之后，会在当前节点的目录下产生很多文件

[ceph-admin@ceph-node-005085 ceph-cluster]$ ll
total 152
-rw------- 1 ceph-admin ceph-admin 71 Jul 22 16:53 ceph.bootstrap-mds.keyring
-rw------- 1 ceph-admin ceph-admin 71 Jul 22 16:53 ceph.bootstrap-mgr.keyring
-rw------- 1 ceph-admin ceph-admin 71 Jul 22 16:53 ceph.bootstrap-osd.keyring
-rw------- 1 ceph-admin ceph-admin 71 Jul 22 16:53 ceph.bootstrap-rgw.keyring
-rw------- 1 ceph-admin ceph-admin 63 Jul 22 16:53 ceph.client.admin.keyring
drwxrwxr-x 2 ceph-admin ceph-admin 4096 Jul 29 14:37 ceph-cluster
-rw-rw-r-- 1 ceph-admin ceph-admin 295 Jul 29 14:59 ceph.conf
-rw-rw-r-- 1 ceph-admin ceph-admin 112788 Jul 29 14:59 ceph-deploy-ceph.log
-rw------- 1 ceph-admin ceph-admin 73 Jul 29 14:59 ceph.mon.keyring
drwxrwxr-x 2 ceph-admin ceph-admin 4096 Jul 22 13:55 iplist

[ceph-admin@ceph-node-005085 ceph-cluster]$

 

2.2 修改ceph.conf，添加下面的内容
public_network=10.10.5.0/24

[ceph-admin@ceph-node-005085 ceph-cluster]$ more ceph.conf
[global]
fsid = 4b597f25-8a49-4bf3-8ead-adfe1f70c9ae
mon_initial_members = ceph-node-005084, ceph-node-005070, ceph-node-005079
mon_host = 10.10.5.84,10.10.5.70,10.10.5.79
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

注意ceph.conf 里面的mon_initial_members，如果主机名包含了.，这个地方的主机名，只会取.前面的字符short hostname，需要在/etc/hosts里面手动绑定

 

2.3 执行ceph-deploy mon create-initial，会在三个节点部署mon，把ceph.conf推送到三个节点上的/etc/ceph/ceph.conf
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create-initial
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe41f233f38>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mon at 0x7fe41f6a78c0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] keyrings : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node-005084 ceph-node-005070 ceph-node-005079
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node-005084 ...
The authenticity of host 'ceph-node-005084 (10.10.5.84)' can't be established.
ECDSA key fingerprint is SHA256:5sfU2vhfTXZWUnyO5ldmrgfhrw7rXcPmyYShLSEzarI.
ECDSA key fingerprint is MD5:0e:5f:6b:d7:fc:dc:16:e2:80:56:34:9f:da:a4:45:86.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph-node-005084' (ECDSA) to the list of known hosts.
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.5.1804 Core
[ceph-node-005084][DEBUG ] determining if provided host has same hostname in remote
[ceph-node-005084][DEBUG ] get remote short hostname
[ceph-node-005084][DEBUG ] deploying mon to ceph-node-005084
[ceph-node-005084][DEBUG ] get remote short hostname
[ceph-node-005084][DEBUG ] remote hostname: ceph-node-005084
[ceph-node-005084][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node-005084][DEBUG ] create the mon path if it does not exist
[ceph-node-005084][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-node-005084/done
[ceph-node-005084][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-node-005084/done
[ceph-node-005084][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-node-005084.mon.keyring
[ceph-node-005084][DEBUG ] create the monitor keyring file
[ceph-node-005084][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-node-005084 --keyring /var/lib/ceph/tmp/ceph-ceph-node-005084.mon.keyring --setuser 167 --setgroup 167
[ceph-node-005084][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-node-005084.mon.keyring
[ceph-node-005084][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node-005084][DEBUG ] create the init path if it does not exist
[ceph-node-005084][INFO ] Running command: sudo systemctl enable ceph.target
[ceph-node-005084][INFO ] Running command: sudo systemctl enable ceph-mon@ceph-node-005084
[ceph-node-005084][WARNIN] Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-node-005084.service to /usr/lib/systemd/system/ceph-mon@.service.
[ceph-node-005084][INFO ] Running command: sudo systemctl start ceph-mon@ceph-node-005084
[ceph-node-005084][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node-005084.asok mon_status
[ceph-node-005084][DEBUG ] ********************************************************************************
[ceph-node-005084][DEBUG ] status for monitor: mon.ceph-node-005084
[ceph-node-005084][DEBUG ] {
[ceph-node-005084][DEBUG ] "election_epoch": 0,
[ceph-node-005084][DEBUG ] "extra_probe_peers": [
[ceph-node-005084][DEBUG ] "10.10.5.70:6789/0",
[ceph-node-005084][DEBUG ] "10.10.5.79:6789/0"
[ceph-node-005084][DEBUG ] ],
[ceph-node-005084][DEBUG ] "feature_map": {
[ceph-node-005084][DEBUG ] "mon": {
[ceph-node-005084][DEBUG ] "group": {
[ceph-node-005084][DEBUG ] "features": "0x3ffddff8eeacfffb",
[ceph-node-005084][DEBUG ] "num": 1,
[ceph-node-005084][DEBUG ] "release": "luminous"
[ceph-node-005084][DEBUG ] }
[ceph-node-005084][DEBUG ] }
[ceph-node-005084][DEBUG ] },
[ceph-node-005084][DEBUG ] "features": {
[ceph-node-005084][DEBUG ] "quorum_con": "0",
[ceph-node-005084][DEBUG ] "quorum_mon": [],
[ceph-node-005084][DEBUG ] "required_con": "0",
[ceph-node-005084][DEBUG ] "required_mon": []
[ceph-node-005084][DEBUG ] },
[ceph-node-005084][DEBUG ] "monmap": {
[ceph-node-005084][DEBUG ] "created": "2019-07-29 15:02:37.698877",
[ceph-node-005084][DEBUG ] "epoch": 0,
[ceph-node-005084][DEBUG ] "features": {
[ceph-node-005084][DEBUG ] "optional": [],
[ceph-node-005084][DEBUG ] "persistent": []
[ceph-node-005084][DEBUG ] },
[ceph-node-005084][DEBUG ] "fsid": "4b597f25-8a49-4bf3-8ead-adfe1f70c9ae",
[ceph-node-005084][DEBUG ] "modified": "2019-07-29 15:02:37.698877",
[ceph-node-005084][DEBUG ] "mons": [
[ceph-node-005084][DEBUG ] {
[ceph-node-005084][DEBUG ] "addr": "10.10.5.84:6789/0",
[ceph-node-005084][DEBUG ] "name": "ceph-node-005084",
[ceph-node-005084][DEBUG ] "public_addr": "10.10.5.84:6789/0",
[ceph-node-005084][DEBUG ] "rank": 0
[ceph-node-005084][DEBUG ] },
[ceph-node-005084][DEBUG ] {
[ceph-node-005084][DEBUG ] "addr": "0.0.0.0:0/1",
[ceph-node-005084][DEBUG ] "name": "ceph-node-005070",
[ceph-node-005084][DEBUG ] "public_addr": "0.0.0.0:0/1",
[ceph-node-005084][DEBUG ] "rank": 1
[ceph-node-005084][DEBUG ] },
[ceph-node-005084][DEBUG ] {
[ceph-node-005084][DEBUG ] "addr": "0.0.0.0:0/2",
[ceph-node-005084][DEBUG ] "name": "ceph-node-005079",
[ceph-node-005084][DEBUG ] "public_addr": "0.0.0.0:0/2",
[ceph-node-005084][DEBUG ] "rank": 2
[ceph-node-005084][DEBUG ] }
[ceph-node-005084][DEBUG ] ]
[ceph-node-005084][DEBUG ] },
[ceph-node-005084][DEBUG ] "name": "ceph-node-005084",
[ceph-node-005084][DEBUG ] "outside_quorum": [
[ceph-node-005084][DEBUG ] "ceph-node-005084"
[ceph-node-005084][DEBUG ] ],
[ceph-node-005084][DEBUG ] "quorum": [],
[ceph-node-005084][DEBUG ] "rank": 0,
[ceph-node-005084][DEBUG ] "state": "probing",
[ceph-node-005084][DEBUG ] "sync_provider": []
[ceph-node-005084][DEBUG ] }
[ceph-node-005084][DEBUG ] ********************************************************************************
[ceph-node-005084][INFO ] monitor: mon.ceph-node-005084 is running
[ceph-node-005084][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node-005084.asok mon_status
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node-005070 ...

执行完后，去对应设备上看，会产生一个进程
/usr/bin/ceph-mon -f --cluster ceph --id ceph-node-005084 --setuser ceph --setgroup ceph

截下来执行ceph -s，会有个报错
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph -s
2019-07-29 15:19:14.044638 7f09dc54a700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-29 15:19:14.044646 7f09dc54a700 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication
2019-07-29 15:19:14.044647 7f09dc54a700 0 librados: client.admin initialization error (2) No such file or directory
[errno 2] error connecting to the cluster

2.4 执行如下命令，将其分发到设备，但是权限是root，ceph-deploy admin ceph-node-005084 ceph-node-005070 ceph-node-005079
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy admin ceph-node-005084 ceph-node-005070 ceph-node-005079
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy admin ceph-node-005084 ceph-node-005070 ceph-node-005079
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f65c7de2290>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['ceph-node-005084', 'ceph-node-005070', 'ceph-node-005079']
[ceph_deploy.cli][INFO ] func : <function admin at 0x7f65c86826e0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node-005084
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph-node-005084][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node-005070
[ceph-node-005070][DEBUG ] connection detected need for sudo
[ceph-node-005070][DEBUG ] connected to host: ceph-node-005070
[ceph-node-005070][DEBUG ] detect platform information from remote host
[ceph-node-005070][DEBUG ] detect machine type
[ceph-node-005070][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node-005079
[ceph-node-005079][DEBUG ] connection detected need for sudo
[ceph-node-005079][DEBUG ] connected to host: ceph-node-005079
[ceph-node-005079][DEBUG ] detect platform information from remote host
[ceph-node-005079][DEBUG ] detect machine type
[ceph-node-005079][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

2.5 再次执行sudo ceph -s 查看集群信息
[ceph-admin@ceph-node-005084 ~]$ ls -lh /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 63 Jul 29 15:26 /etc/ceph/ceph.client.admin.keyring
[ceph-admin@ceph-node-005084 ~]$ sudo ceph -s
cluster:
id: 4b597f25-8a49-4bf3-8ead-adfe1f70c9ae
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph-node-005070,ceph-node-005079,ceph-node-005084
mgr: no daemons active
osd: 0 osds: 0 up, 0 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0B
usage: 0B used, 0B / 0B avail
pgs:

检查集群状态
[ceph-admin@ceph-node-005084 ~]$ ceph health
HEALTH_OK

3.1 给这3台设备分别安装mgr
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy mgr create ceph-node-005084
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy mgr create ceph-node-005084
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] mgr : [('ceph-node-005084', 'ceph-node-005084')]
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f29368d5d40>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mgr at 0x7f2936d295f0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-node-005084:ceph-node-005084
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-node-005084
[ceph-node-005084][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node-005084][WARNIN] mgr keyring does not exist yet, creating one
[ceph-node-005084][DEBUG ] create a keyring file
[ceph-node-005084][DEBUG ] create path recursively if it doesn't exist
[ceph-node-005084][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-node-005084 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-node-005084/keyring
[ceph-node-005084][INFO ] Running command: sudo systemctl enable ceph-mgr@ceph-node-005084
[ceph-node-005084][INFO ] Running command: sudo systemctl start ceph-mgr@ceph-node-005084
[ceph-node-005084][INFO ] Running command: sudo systemctl enable ceph.target
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy mgr create ceph-node-005070
……
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy mgr create ceph-node-005079
……

4.1 增加SOD，列出磁盘信息
查看设备的磁盘信息
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy disk list ceph-node-005084
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk list ceph-node-005084
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4600473a28>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : ['ceph-node-005084']
[ceph_deploy.cli][INFO ] func : <function disk at 0x7f46008cede8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph-node-005084][INFO ] Running command: sudo fdisk -l
[ceph-node-005084][INFO ] Disk /dev/sdb: 4000.8 GB, 4000787030016 bytes, 7814037168 sectors
[ceph-node-005084][INFO ] Disk /dev/sda: 4000.8 GB, 4000787030016 bytes, 7814037168 sectors

4.2 这里需要登陆设备上确认，/dev/sdb是空闲的，没有使用
[ceph-admin@ceph-node-005084 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 3.7T 0 disk
├─sda1 8:1 0 2M 0 part
├─sda2 8:2 0 488M 0 part /boot
├─sda3 8:3 0 558.3G 0 part /
└─sda4 8:4 0 3.1T 0 part /data
sdb 8:16 0 3.7T 0 disk

4.3 清理磁盘分区信息
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy disk zap ceph-node-005084 /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk zap ceph-node-005084 /dev/sdb
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : zap
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe3ea74ea28>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : ceph-node-005084
[ceph_deploy.cli][INFO ] func : <function disk at 0x7fe3eaba9de8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : ['/dev/sdb']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdb on ceph-node-005084
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.5.1804 Core
[ceph-node-005084][DEBUG ] zeroing last few blocks of device
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph-node-005084][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdb
[ceph-node-005084][DEBUG ] --> Zapping: /dev/sdb
[ceph-node-005084][DEBUG ] --> --destroy was not specified, but zapping a whole device will remove the partition table
[ceph-node-005084][DEBUG ] Running command: wipefs --all /dev/sdb
[ceph-node-005084][DEBUG ] stdout: /dev/sdb: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
[ceph-node-005084][DEBUG ] Running command: dd if=/dev/zero of=/dev/sdb bs=1M count=10
[ceph-node-005084][DEBUG ] stderr: 10+0 records in
[ceph-node-005084][DEBUG ] 10+0 records out
[ceph-node-005084][DEBUG ] 10485760 bytes (10 MB) copied
[ceph-node-005084][DEBUG ] stderr: , 0.0119554 s, 877 MB/s
[ceph-node-005084][DEBUG ] --> Zapping successful for: <Raw Device: /dev/sdb>

4.4此时查看集群信息
[ceph-admin@ceph-node-005084 ~]$ sudo ceph -s
cluster:
id: 8ac4193c-f1bb-4dcb-821f-e290f1e4be82
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph-node-005070,ceph-node-005079,ceph-node-005084
mgr: ceph-node-005084(active), standbys: ceph-node-005079
osd: 0 osds: 0 up, 0 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0B
usage: 0B used, 0B / 0B avail
pgs:

4.5 查看osd tree
[ceph-admin@ceph-node-005084 ~]$ sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0 root default_release

5.1 启动dashboard
[ceph-admin@ceph-node-005084 ~]$ sudo ceph mgr module enable dashboard
5.2.绑定开启dashboard模块的ceph-mgr节点的地址
[ceph-admin@ceph-node-005084 ~]$ sudo ceph config-key set mgr/dashboard/server_addr 10.10.5.84
set mgr/dashboard/ceph-node-005084/server_addr
5.3 查看监听的端口
[ceph-admin@ceph-node-005084 ~]$ netstat -ant|grep 7000
tcp 0 0 10.10.5.84:7000 0.0.0.0:* LISTEN
tcp6 0 0 ::1:48292 ::1:7000 TIME_WAIT
tcp6 0 0 ::1:48236 ::1:7000 TIME_WAIT
5.4 修改端口
[ceph-admin@ceph-node-005070 ~]$ sudo ceph config-key set mgr/dashboard/server_port 8088
5.5 重启一下服务
sudo ceph mgr module disable dashboard
sudo ceph mgr module enable dashboard
5.6 再次查看端口
[ceph-admin@ceph-node-005084 ~]$ sudo netstat -tunpl | grep ceph-mgr
tcp 0 0 10.10.5.84:6800 0.0.0.0:* LISTEN 1628868/ceph-mgr
tcp 0 0 10.10.5.84:8088 0.0.0.0:* LISTEN 1628868/ceph-mgr

 

6.1 创建OSD
ceph-deploy: error: unrecognized arguments: --overwrite-conf
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy --overwrite-conf osd create ceph-node-005084 --fs-type btrfs --data /dev/sdb

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy --overwrite-conf osd create ceph-node-005084 --fs-type btrfs --data /dev/sdb
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] bluestore : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe3a9072b48>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : btrfs
[ceph_deploy.cli][INFO ] block_wal : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] journal : None
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] host : ceph-node-005084
[ceph_deploy.cli][INFO ] filestore : None
[ceph_deploy.cli][INFO ] func : <function osd at 0x7fe3a94c6d70>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.cli][INFO ] data : /dev/sdb
[ceph_deploy.cli][INFO ] block_db : None
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node-005084
[ceph-node-005084][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node-005084][WARNIN] osd keyring does not exist yet, creating one
[ceph-node-005084][DEBUG ] create a keyring file
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph-node-005084][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[ceph-node-005084][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key
[ceph-node-005084][DEBUG ] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new dd82cc52-abee-4011-8ade-408cebb34ac7
[ceph-node-005084][DEBUG ] Running command: vgcreate --force --yes ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870 /dev/sdb
[ceph-node-005084][DEBUG ] stdout: Physical volume "/dev/sdb" successfully created.
[ceph-node-005084][DEBUG ] stdout: Volume group "ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870" successfully created
[ceph-node-005084][DEBUG ] Running command: lvcreate --yes -l 100%FREE -n osd-block-dd82cc52-abee-4011-8ade-408cebb34ac7 ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870
[ceph-node-005084][DEBUG ] stdout: Logical volume "osd-block-dd82cc52-abee-4011-8ade-408cebb34ac7" created.
[ceph-node-005084][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key
[ceph-node-005084][DEBUG ] Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
[ceph-node-005084][DEBUG ] Running command: restorecon /var/lib/ceph/osd/ceph-0
[ceph-node-005084][DEBUG ] Running command: chown -h ceph:ceph /dev/ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870/osd-block-dd82cc52-abee-4011-8ade-408cebb34ac7
[ceph-node-005084][DEBUG ] Running command: chown -R ceph:ceph /dev/dm-0
[ceph-node-005084][DEBUG ] Running command: ln -s /dev/ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870/osd-block-dd82cc52-abee-4011-8ade-408cebb34ac7 /var/lib/ceph/osd/ceph-0/block
[ceph-node-005084][DEBUG ] Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
[ceph-node-005084][DEBUG ] stderr: got monmap epoch 1
[ceph-node-005084][DEBUG ] Running command: ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQBmVkBdpC+FKxAAqI0BDX/OtgO9uBdlHpX0YQ==
[ceph-node-005084][DEBUG ] stdout: creating /var/lib/ceph/osd/ceph-0/keyring
[ceph-node-005084][DEBUG ] stdout: added entity osd.0 auth auth(auid = 18446744073709551615 key=AQBmVkBdpC+FKxAAqI0BDX/OtgO9uBdlHpX0YQ== with 0 caps)
[ceph-node-005084][DEBUG ] Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
[ceph-node-005084][DEBUG ] Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
[ceph-node-005084][DEBUG ] Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid dd82cc52-abee-4011-8ade-408cebb34ac7 --setuser ceph --setgroup ceph
[ceph-node-005084][DEBUG ] --> ceph-volume lvm prepare successful for: /dev/sdb
[ceph-node-005084][DEBUG ] Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[ceph-node-005084][DEBUG ] Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870/osd-block-dd82cc52-abee-4011-8ade-408cebb34ac7 --path /var/lib/ceph/osd/ceph-0
[ceph-node-005084][DEBUG ] Running command: ln -snf /dev/ceph-df1d17e4-6d0f-4ae8-bf9d-54b2b4746870/osd-block-dd82cc52-abee-4011-8ade-408cebb34ac7 /var/lib/ceph/osd/ceph-0/block
[ceph-node-005084][DEBUG ] Running command: chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
[ceph-node-005084][DEBUG ] Running command: chown -R ceph:ceph /dev/dm-0
[ceph-node-005084][DEBUG ] Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[ceph-node-005084][DEBUG ] Running command: systemctl enable ceph-volume@lvm-0-dd82cc52-abee-4011-8ade-408cebb34ac7
[ceph-node-005084][DEBUG ] stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-dd82cc52-abee-4011-8ade-408cebb34ac7.service to /usr/lib/systemd/system/ceph-volume@.service.
[ceph-node-005084][DEBUG ] Running command: systemctl enable --runtime ceph-osd@0
[ceph-node-005084][DEBUG ] stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service to /usr/lib/systemd/system/ceph-osd@.service.
[ceph-node-005084][DEBUG ] Running command: systemctl start ceph-osd@0
[ceph-node-005084][DEBUG ] --> ceph-volume lvm activate successful for osd ID: 0
[ceph-node-005084][DEBUG ] --> ceph-volume lvm create successful for: /dev/sdb
[ceph-node-005084][INFO ] checking OSD status...
[ceph-node-005084][DEBUG ] find the location of an executable
[ceph-node-005084][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host ceph-node-005084 is now ready for osd use.

 

6.2 添加元数据服务器
至少需要一个元数据服务器才能使用 CephFS ，执行下列命令创建元数据服务器：
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-node-005084:ceph-node-005084
[ceph-node-005084][DEBUG ] connection detected need for sudo
[ceph-node-005084][DEBUG ] connected to host: ceph-node-005084
[ceph-node-005084][DEBUG ] detect platform information from remote host
[ceph-node-005084][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-node-005084
[ceph-node-005084][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node-005084][WARNIN] mds keyring does not exist yet, creating one
[ceph-node-005084][DEBUG ] create a keyring file
[ceph-node-005084][DEBUG ] create path if it doesn't exist
[ceph-node-005084][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-node-005084 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-node-005084/keyring
[ceph-node-005084][INFO ] Running command: sudo systemctl enable ceph-mds@ceph-node-005084
[ceph-node-005084][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-node-005084.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-node-005084][INFO ] Running command: sudo systemctl start ceph-mds@ceph-node-005084
[ceph-node-005084][INFO ] Running command: sudo systemctl enable ceph.target

到此，可以创建RBD和cephFS的ceph集群搭建完成。
[ceph-admin@ceph-node-005084 ~]$ sudo ceph -s
cluster:
id: 8ac4193c-f1bb-4dcb-821f-e290f1e4be82
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph-node-005070,ceph-node-005079,ceph-node-005084
mgr: ceph-node-005084(active), standbys: ceph-node-005079
osd: 7 osds: 7 up, 7 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0B
usage: 7.03GiB used, 25.5TiB / 25.5TiB avail
pgs:

=======================================

常用技巧
添加一个新的osd
推送配置文件
ceph-deploy --overwrite-conf config push ceph-node-005088
ceph-deploy admin ceph-node-005088
查看磁盘信息
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy disk list ceph-node-005088
清理磁盘分区信息
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy disk zap ceph-node-005088 /dev/sdb
创建osd
[ceph-admin@ceph-node-005085 ceph-cluster]$ ceph-deploy --overwrite-conf osd create ceph-node-005088 --fs-type btrfs --data /dev/sdb

 

 

报错处理：

[ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

1.创建目录

sudo mkdir /var/run/ceph/

sudo chown ceph.ceph /var/run/ceph/

sudo chmod 755  /var/run/ceph/

重启服务  sudo systemctl restart ceph-mon@-preonline120001

 

如果部署过程中出错，想重新执行，可以执行如下命令进行清理

ceph-deploy purgedata node1 ndoe2

ceph-deploy forgetkeys
sudo rm -rf /etc/ceph/*
sudo rm -rf /var/lib/ceph/*/*
sudo rm -rf /var/log/ceph/*
sudo rm -rf /var/run/ceph/*