生产级别的Ceph集群搭建

准备环境

主机名 业务IP 集群IP 磁盘规划 功能介绍
ceph-deploy 172.16.143.110 172.16.138.110 200/1 管理Ceph集群
ceph-mon1 172.16.143.111 172.16.138.131 200/1 监控Ceph集群状态,拓扑等信息
ceph-mon2 172.16.143.112 172.16.138.132 200/1 监控Ceph集群状态,拓扑等信息
ceph-mon3 172.16.143.113 172.16.138.113 200/1 监控Ceph集群状态,拓扑等信息
ceph-mgr1 172.16.143.114 172.16.138.114 200/1 Ceph 集群的管理,实现统一接口
ceph-mgr2 172.16.143.115 172.16.138.115 200/1 Ceph 集群的管理,实现统一接口
ceph-node1 172.16.143.116 172.16.138.116 200/1 100/2 Ceph 集群的存储节点
ceph-node2 172.16.143.117 172.16.138.117 200/1 100/2 Ceph 集群的存储节点
ceph-node3 172.16.143.118 172.16.138.118 200/1 100/2 Ceph 集群的存储节点
ceph-node4 172.16.143.119 172.16.138.119 200/1 100/2 Ceph 集群的存储节点
  • 配置2张网卡,一个是业务访问使用,一个是集群内部访问使用
  • 关闭防火墙
  • 关闭selinux
  • 配置时间同步
  • 配置hosts

配置Ceph的源

 wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add -
sudo apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific/ bionic main'
sudo apt update

 配置ceph用户

groupadd -r -g 2022 ceph && useradd -r -m -s /bin/bash -u 2022 -g 2022 ceph && echo ceph:123456 | chpasswd
echo "ceph ALL=(ALL)   NOPASSWD:ALL" >> /etc/sudoers

配置ceph-deploy节点免密登录

 ssh-keygen -t rsa
 ssh-copy-id -i .ssh/id_rsa.pub ceph@172.16.143.110
...
...
 ssh-copy-id -i .ssh/id_rsa.pub ceph@172.16.143.119

安装Ceph的部署工具ceph-deploy

sudo apt install  ceph-deploy

安装Ceph

初始化mon

mkdir ceph-cluster
cd ceph-cluster

生成配置文件(ceph-deploy节点)

$ ceph-deploy new --cluster-network 172.16.138.0/24 --public-network 172.16.143.0/24 ceph-mon1
$ ls
ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring

安装ceph-mon(ceph-moin)

$ sudo apt install ceph-mon -y

 

 回车

 

 回车

初始化mon

ceph-deploy mon create-initial

 

 验证mon进程是否启动

$ ps -ef |grep mon
message+   757     1  0 04:40 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root       802     1  0 04:40 ?        00:00:00 /usr/lib/accountsservice/accounts-daemon
daemon     832     1  0 04:40 ?        00:00:00 /usr/sbin/atd -f
ceph      6739     1  0 05:14 ?        00:00:00 /usr/bin/ceph-mon -f --cluster ceph --id ceph-mon1 --setuser ceph --setgroup ceph
root      7343  1566  0 05:16 pts/0    00:00:00 grep --color=auto mon

安装ceph-common及分发ceph-admin管理秘钥

//root用户执行
$ apt install ceph-common -y
$ ceph-deploy admin ceph-deploy
$ sudo chown ceph:ceph /etc/ceph/ceph.client.admin.keyring
$ ceph -s
  cluster:
    id:     6e278817-8019-4a06-82b3-b4d24d7dd743
    health: HEALTH_WARN
            mon is allowing insecure global_id reclaim

  services:
    mon: 1 daemons, quorum ceph-mon1 (age 60m)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

安装ceph-mgr

mgr节点执行

 apt install ceph-mgr -y

deploy节点执行

ceph-deploy mgr create  ceph-mgr1

 

安装ceph-node

deploy节点执行

//分别执行1,2,3,4节点
ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node4

 擦除远端磁盘(node节点的都要执行,sdb  sdc)

ceph-deploy disk zap ceph-node1 /dev/sdb

添加主机OSD(node节点的所有磁盘都要执行)

ceph-deploy osd create ceph-node2 --data /dev/sdb

查看集群状态

$ ceph -s
  cluster:
    id:     6e278817-8019-4a06-82b3-b4d24d7dd743
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-mon1 (age 3d)
    mgr: ceph-mgr1(active, since 3d)
    osd: 8 osds: 8 up (since 9s), 8 in (since 17s)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   45 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

 扩展mon服务

mon节点2/3 安装ceph-mon

$ apt install ceph-mon

 添加mon

$ ceph-deploy mon add ceph-mon3
$  ceph -s
  cluster:
    id:     6e278817-8019-4a06-82b3-b4d24d7dd743
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 29s)
    mgr: ceph-mgr1(active, since 3d)
    osd: 8 osds: 8 up (since 2h), 8 in (since 2h)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   45 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

查看mon状态

$ ceph quorum_status --format json-pretty

扩展mgr

mgr节点安装mgr

$ apt install ceph-mgr

添加mgr

$ ceph-deploy mgr create ceph-mgr2

查看状态

$ ceph -s
  cluster:
    id:     6e278817-8019-4a06-82b3-b4d24d7dd743
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 5m)
    mgr: ceph-mgr1(active, since 3d), standbys: ceph-mgr2
    osd: 8 osds: 8 up (since 2h), 8 in (since 2h)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   45 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

 

posted @ 2021-08-24 14:16  大胖猴  阅读(698)  评论(0编辑  收藏  举报