ceph centos7 安装

参考文章：http://docs.ceph.com/docs/master/start/quick-ceph-deploy/#storing-retrieving-object-data

重写conf文件

ceph-deploy --overwrite-conf config push ceph-admin ceph-node1 ceph-node2

注：

1. 所有操作均可在ceph-deploy操作
2. cepy-deploy主机需要与其他主机进行免密登陆
3. 所有主机均要保持时间一致
4. 参考官网文档，会导致包下载不了

一.集群安装

一.初始安装

安装cepy-deploy:

yum install http://mirrors.163.com/ceph/rpm-luminous/el7/noarch/ceph-deploy-2.0.0-0.noarch.rpm

mkdir /opt/ceph
cd /opt/ceph

注意：如有中断重来全新安装，可做如下操作：

ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*

初始化一个monitor节点

ceph-deploy new {initial-monitor-nodes}
ceph-deploy new node1

并修改工作目录下的ceph.conf文件,添加如下：

public_network = {ip-address}/{bits}

note: to use IPs in the 10.1.2.0/24 (or 10.1.2.0/255.255.255.0) network

安装ceph运行的二进制包，如有新节点，则需要对节点执行该操作

ceph-deploy install --release luminous --repo-url http://mirrors.163.com/ceph/rpm-luminous/el7 --gpg-url http://mirrors.163.com/ceph/keys/release.asc c7-node1 c7-node2 c7-node3

初始化mon：

ceph-deploy mon create-initial

分发密钥文件

ceph-deploy admin {ceph-nodes}
ceph-deploy admin node1 node2 node3

部署mgr(manager)进程

ceph-deploy mgr create node1
ceph mgr module enable dashboard    启动dashboard界面

默认web页面的端口：7000

添加osd磁盘

ceph-deploy osd create --data /dev/sdb node1
ceph-deploy osd create --data /dev/sdb node2
ceph-deploy osd create --data /dev/sdb node3

检查集群状态

ceph health
ceph -s

二.扩展集群

添加metadata，如果不用cephfs，则可以跳过此步

ceph-deploy mds create {ceph-node}
ceph-deploy mds create node1

添加monitors
A Ceph Storage Cluster Requires At Least One Ceph Monitor And Ceph Manager To Run. For High Availability, Ceph Storage Clusters Typically Run Multiple Ceph Monitors So That The Failure Of A Single Ceph Monitor Will Not Bring Down The Ceph Storage Cluster. Ceph Uses The Paxos Algorithm, Which Requires A Majority Of Monitors (I.E., Greather Than N/2 Where N Is The Number Of Monitors) To Form A Quorum. Odd Numbers Of Monitors Tend To Be Better, Although This Is Not Required.

修改工作目录下的ceph.conf文件，修改mon_initial_mermbers和mon_host，并添加public_network项。并执行如下命令。重写配置文件

ceph-deploy --overwrite-conf config push c7-node1 c7-node2 c7-node3

然后在进行添加mon

ceph-deploy mon add {ceph-nodes}
ceph-deploy mon add node2 node3

添加mgr

ceph-deploy mgr create node2 node3

添加网关实例

ceph-deploy rgw create {gateway-node}
ceph-deploy rgw create node1

web端口验证：7480

三.上传数据实例

echo akfbka > testfile.txt
ceph osd pool create mytest 8
rados put test-object-1 testfile.txt --pool=mytest

rados -p mytest ls

二.部署MDS

创建mds

ceph-deploy --overwrite-conf mds create c7-node1

创建pool

mds需要两个pool，一个pool存储数据，一个pool存储元数据。因此创建fs_data和fs_metadata两个存储池。

ceph osd pool create fs_data 3
ceph osd pool create fs_metadata 3

创建一个cephFS

命令格式：ceph fs new <fs_name> <metadata> <data>
ceph fs new cephfs fs_metadata fs_data

查看创建测cephfs

ceph fs ls          #查看

ceph mds stat       #查看状态

挂载

两种方式： kelnel Driver和FUSE

kelnel方式(未能成功)

mkdir /cephfs_test
mount -t ceph [IP]:6789:/ /cephfs_test

开机自动挂载，在/etc/fstab中添加如下：

[ip]:6789://cephfs_test ceph noatime 0 2

fuse方式
通过挂载fuse的方式挂载

yum instal ceph-fuse
mkdir /fuse_test
ceph-fuse -m [ip]:6789 /fuse_test

开机自动挂载，在/etc/fstab中添加如下（待验证）：

id=admin, conf=/etc/ceph/ceph.conf/fuse_test fuse.ceph defaults 0 0

注：id是用户账号

三.块存储

加载rbd模块

modprobe rbd            # 加载rbd
modinfo rbd             # 查看模块信息

创建一个10GB大小的块设备

rbd create test_image --size 10240      # 创建
rbd list                                # 查看
rbd info test_image

将test_image块设备映射到操作系统

rbd map test_image

查看系统中已经映射的块设备

rbd showmapped

取消块设备映射

rbd unmap /dev/rbd0

rbd常见操作

创建存储池

rados mkpool rbd_pool       创建名为rbd_pool存储池
rados lspools               查看所有的pool

RBD IMAGE
在pool中创建一个大小为1GB的image

rbd create rbd_pool/image1 --size 1024 --image-format 2

注：创建image时，设置格式为2。format 1是原始格式，也是创建image的默认格式，format2支持rbd分层，是实现COW(Copy-On-Write)的前提

查看存储中的rbd

rbd ls rbd_pool
rbd info rbd_pool/image1

注：ceph集群中一个object对象默认大小是4MB，也可以指定object大小。

rbd create rbd_pool/image2 --size 1024 --order 24 --image-format 2

注：--order表示指定object大小为2的24次方，即16MB
4. 删除image

rbd rm rbd_pool/image2

快照、克隆

略

image导入导出

略

四.对象存储

注：ceph的Firely版本开始，新的RGW前度Civeweb已经引入，是一个轻量级的web服务器，viceweb已经被嵌入到ceph-radosgw服务。是的ceph对象存储服务更快，更容易。

创建keyring:

cd /etc/ceph/
ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring        #创建keyring
chmod +r /etc/ceph.client.radosgw.keyring

为RGW实例生成网关用户和密钥，此处RGW实例名为gatewat

ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key

给密钥添加权限：

ceph-authtool -n client.radosgw.gateway --cap osd "allow rwx" --cap mon "allow rwx" /etc/ceph/ceph.client.radosgw.keyring

将密钥添加到ceph集群中

ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring

将密钥分配到ceph RGW节点

scp /etc/ceph/ceph.client.radosgw.keyring node2:/etc/ceph/ceph.client.radosgw.keyring

修改配置文件

在/opt/ceph/ceph.conf添加如下参数：

[client.radosgw.gateway]
host = node1,node2,node3
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw dns name = cephclient001.flybear.com.cn             #此处对我dns域名
rgw print continue = false

注：如果实例名不是gateway的话，需要做相应改变
推送配置文件

cd /opt/ceph/
ceph-deploy --overwrite-conf rgw create node1 node2 node3

验证：curl http://node1:7480

使用s3客户端访问

确保可以访问ceph集群：

ceph -s -k /etc/ceph/ceph.client.radosgw.keyring --name client.radosgw.gateway

注意：以下操作来自网友
为s3创建rados网关用户

radosgw-admin user create --uid=jiangdb --display-name="jiangdb" --email=jiangdianbin@126.com -k /etc/ceph/ceph.client.radosgw.keyring --name client.radosgw.gateway

输出如下：

{
"user_id": "jiangdb",
"display_name": "jiangdb",
"email": "jiangdianbin@126.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "jiangdb",
"access_key": "0LJC2NRCXJ3GO31OWZPA",
"secret_key": "TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}

说明：需要两个密钥记录一下：
"access_key": 0LJC2NRCXJ3GO31OWZPA
"secret_key": TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq

搭建下DNS服务器，搭建在rgw-node1上

方法一、搭建DNS
（此处没有用，测试使用hosts了。生产时是需要DNS的）
方法二、使用/etc/hosts本地解析

安装pip软件并升级

yum install python-pip
pip install --upgrade pip

安装s3cmd并配置

安装：下载s3cmd软件在这个http://s3tools.org/download下载，并解压然后安装s3cmd-master.zip，然后进入目录后执行：

pip install s3cmd           # pip安装
s3cmd --configure           # s3配置

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 0LJC2NRCXJ3GO31OWZPA
Secret Key: TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint []:cephclient001.flybear.com.cn：7480

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []:cephclient001.flybear.com.cn:7480

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password [:]   
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol []:No

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
Access Key: 0LJC2NRCXJ3GO31OWZPA
Secret Key: TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq
Default Region: US
S3 Endpoint: cephclient001.flybear.com.cn:7480
DNS-style bucket+hostname:port template for accessing a bucket: cephclient001.flybear.com.cn:7480
Encryption password: 
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name: 
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
[root@cephclient001 s3cmd-master]#
（也可以手动编辑/root/.s3cfg配置文件）

测试：使用客户端s3cmd链接并管理对象存储资源创建Bucket，并将一个文件放上去

新建Bucket

s3cmd mb s3://test1
Bucket 's3://test1/' created

查看现有Bucket

s3cmd  ls
2017-10-26 12:02  s3://test1

上传Object文件

s3cmd put setup.cfg s3://test1
s3cmd  ls s3://test1        # 查看

下载Object

cd /tmp
s3cmd  get s3://test1/setup.cfg

使用Swift API访问对象存储

安装客户端软件

yum install python-setuptools
pip install --upgrade python-setuptools
pip install --upgrade python-swiftclient

在RGW上为Swift接口创建RADOS网关子用户

radosgw-admin subuser create --uid=jiangdb --subuser=jiangdb:swift --access=full -k /etc/ceph/ceph.client.radosgw.keyring --name client.radosgw.gateway

{
"user_id": "jiangdb",
"display_name": "jiangdb",
"email": "jiangdianbin@126.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{
"id": "jiangdb:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "jiangdb",
"access_key": "0LJC2NRCXJ3GO31OWZPA",
"secret_key": "TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq"
}
],
"swift_keys": [
{
"user": "jiangdb:swift",
"secret_key": "9EgCRZORdxSUPgO50OsUnHncQ02LdBzFoSk3b97H"
}
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}

为jiangdb:swift子用户创建密钥

radosgw-admin key create --subuser=jiangdb:swift --key-type=swift --gen-secret -k /etc/ceph/ceph.client.radosgw.keyring --name client.radosgw.gateway

{
"user_id": "jiangdb",
"display_name": "jiangdb",
"email": "jiangdianbin@126.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{
"id": "jiangdb:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "jiangdb",
"access_key": "0LJC2NRCXJ3GO31OWZPA",
"secret_key": "TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq"
}
],
"swift_keys": [
{
"user": "jiangdb:swift",
"secret_key": "HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM"
}
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}

获取swift子用户和密钥

radosgw-admin user info --uid jiangdb

{
"user_id": "jiangdb",
"display_name": "jiangdb",
"email": "jiangdianbin@126.com",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{
"id": "jiangdb:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "jiangdb",
"access_key": "0LJC2NRCXJ3GO31OWZPA",
"secret_key": "TzxxpAEd1fP6gRO3mfRgI1pkGFMGHZMdzB6N28Uq"
}
],
"swift_keys": [
{
"user": "jiangdb:swift",
"secret_key": "HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM"
}
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}

此密码下面要用

"user": "jiangdb:swift",
"secret_key": "HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM"

访问ceph对象存储，列出默认bucket:

"user": "jiangdb:swift",
"secret_key": "HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM"
}
使用子用户的密钥访问存储：

swift -A http://192.168.100.101:7480/auth/1.0 -U jiangdb:swift  -K HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM list

test1

增加一个新的bucket: swift-bucket

swift -A http://192.168.100.101:7480/auth/1.0 -U jiangdb:swift  -K HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM post swift-bucket

列出所有bucket，查看新增的swift-bucket是否存在

swift -A http://192.168.100.101:7480/auth/1.0 -U jiangdb:swift  -K HLoJBtopKBrSUZ7PsbLN32LhiG2HV0sdjmGIxFEM list

swift-bucket
test1

配置、验证完毕，Swift API可以访问对象存储。

posted @ 2021-03-16 17:06 lei&dong 阅读(256) 评论(0) 收藏举报

刷新页面返回顶部

lei.dong