07-ceph存储网关rgw
一、对象存储网关组件rgw概述
1.对象存储网关概述
Ceph对象网关可以将数据存储在用于存储来自cephfs客户端或ceph rbd客户端的数据的同一ceph存储集群中。
object是对象存储系统中数据存储的基本单位，每个Object是数据和数据属性集的综合体，数据可以根据应用的需求进行设置，包括数据分布，服务质量等。
每个对象自我维护其属性，从而简化了存储系统的管理任务，对象的大小可以不同，甚至可以包含整个数据结构，如文件，数据库表项等，文件的上传和下载，默认有一个最大的数据块15MB。
Ceph对象存储使用Ceph对象网关守护进程（Rados GateWay，简称rgw），它是用于与ceph存储集群进行交互式的HTTP服务器。
Ceph RGW基于librados，是为应用提供RESTful类型的对象存储接口，默认使用Civetweb作为其Web Service。
在N版本中Civetweb默认使用法端口7480提供服务，但R版本(18.2.4)中使用了80端口，若想自定义端口就需要修改ceph的配置文件。
- 自0.80版本(Firefly，2014-05-01～2016-04-01)起，Ceph放弃了apache和fastcgi提供radosgw服务;
- 默认嵌入了在ceph-radosgw进程中的Citeweb，这种新的实现方式更加轻便和简洁，但直到Ceph 11.0.1 Kraken(2017-01-01~2017-08-01)版本，Citeweb才开始支持SSL协议。
推荐阅读:
https://docs.ceph.com/en/squid/radosgw/
https://docs.ceph.com/en/nautilus/radosgw/
https://docs.ceph.com/en/nautilus/radosgw/bucketpolicy/
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/bucketnamingrules.html
https://www.s3express.com/help/help.html
2.对象存储系统的核心资源概述
各种存储方案虽然在设计与实现上有所区别，但大多数对象存储系统对外呈现的核心资源类型大同小异。
一般来说，一个对象存储系统的核心资源应该包括（User），存储桶（Bucket）和对象（object），它们之间的关系是:
- 1.User将Object存储到存储系统上的Bucket;
- 2.存储桶属于某个用户并可以容纳对象，一个存储桶用于存储多个对象;
- 3.同一个用户可以拥有多个存储桶，不同用户允许使用相同名称的bucket;
3.ceph rgw支持的接口
RGW需要自己独有的守护进程服务才可以正常的使用，RGW并非必须的接口，仅在需要用到S3和Swift兼容的RESTful接口时才需要部署RGW实例，RGW在创建的时候，会自动初始化自己的存储池。
如上图所示，由于RGW提供与OpenStack Swift和Amazon S3兼容的接口，因此ceph对象网关具有自己的用户管理。
- Amazon S3:
兼容Amazon S3RESTful API，侧重命令行操作。
提供了user，bucket和object分别表示用户，存储桶和对象，其中bucket隶属于user。
因此user名称即可作为bucket的名称空间，不同用户允许使用相同的bucket。
- OpenStack Swift:
兼容OpenStack Swift API，侧重应用代码实践。
提供了user，container和object分别对应于用户，存储桶和对象，不过它还额外为user提供了父及组件account，用于表示一个项目或租户。
因此一个account中可包含一到多个user，它们可共享使用同一组container，并为container提供名称空间。
- RadosGW:
提供了user，subuser，bucket和object，其中user对应于S3的user，而subuser则对应于Swift的user，不过user和subuser都不支持为bucket提供名称空间，因此不同用户的存储桶也不允许同名。
不过，自Jewel版本（10.2.11，2016-04-01～2018-07-01）起，RadosGW引入了tenant（租户）用于为user和bucket提供名称空间，但它是可选组件。
Jewel版本之前，radosgw的所有user位于同一名称空间，它要求所有user的ID必须唯一，并且即便是不同user的bucket也不允许使用相同的bucket ID。
二、部署radosgw实操案例
1.部署之前查看集群状态
[root@ceph141 ~]# ceph -s
cluster:
id: 11e66474-0e02-11f0-82d6-4dcae3d59070
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 20h)
mgr: ceph141.mbakds(active, since 2d), standbys: ceph142.qgifwo
mds: 1/1 daemons up, 1 standby
osd: 9 osds: 9 up (since 20h), 9 in (since 2d); 23 remapped pgs
data:
volumes: 1/1 healthy
pools: 5 pools, 451 pgs
objects: 9.91k objects, 25 GiB
usage: 88 GiB used, 5.3 TiB / 5.3 TiB avail
pgs: 0.222% pgs not active
1373/29736 objects misplaced (4.617%)
427 active+clean
22 active+remapped+backfill_wait
1 peering
1 active+remapped+backfilling
io:
client: 167 KiB/s wr, 0 op/s rd, 6 op/s wr
recovery: 58 MiB/s, 14 objects/s
2.创建一个服务
[root@ceph141 ~]# ceph orch apply rgw dezyanding
Scheduled rgw.dezyanding update...
3.部署rgw组件
[root@ceph141 ~]# ceph orch daemon add rgw dezyanding ceph142
Deployed rgw.dezyanding.ceph142.tmtpzs on host 'ceph142'
4.检查rgw组件是否部署成功
[root@ceph141 ~]# ceph -s
cluster:
id: 11e66474-0e02-11f0-82d6-4dcae3d59070
health: HEALTH_WARN
Reduced data availability: 1 pg peering
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 20h)
mgr: ceph141.mbakds(active, since 2d), standbys: ceph142.qgifwo
mds: 1/1 daemons up, 1 standby
osd: 9 osds: 9 up (since 20h), 9 in (since 2d); 39 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones) # 观察是否有该组件
data:
volumes: 1/1 healthy
pools: 9 pools, 599 pgs
objects: 10.22k objects, 25 GiB
usage: 89 GiB used, 5.3 TiB / 5.3 TiB avail
pgs: 0.167% pgs unknown
0.668% pgs not active
1583/30660 objects misplaced (5.163%)
553 active+clean
37 active+remapped+backfill_wait
4 active+remapped+backfilling
3 peering
1 unknown
1 remapped+peering
io:
client: 3.6 MiB/s rd, 166 MiB/s wr, 195 op/s rd, 2.38k op/s wr
recovery: 59 MiB/s, 20 objects/s
progress:
Global Recovery Event (105s)
[=========================...] (remaining: 8s)
5.查看rgw默认创建的存储池信息
[root@ceph141 ~]# ceph osd pool ls
...
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta
6.查看rgw类型的服务信息及所在的主机列表
[root@ceph141 ~]# ceph orch ls rgw rgw.dezyanding --export
service_type: rgw
service_id: dezyanding
service_name: rgw.dezyanding
placement:
count: 2
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --service_name rgw.dezyanding
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
rgw.dezyanding.ceph141.svisis ceph141 *:80 running (21m) 3m ago 21m 102M - 19.2.1 f2efb0401a30 1d3c29e587dd
rgw.dezyanding.ceph142.tmtpzs ceph142 *:80 running (21m) 3m ago 21m 105M - 19.2.1 f2efb0401a30 4f90102b7675
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --daemon_type rgw
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
rgw.dezyanding.ceph141.svisis ceph141 *:80 running (21m) 3m ago 21m 102M - 19.2.1 f2efb0401a30 1d3c29e587dd
rgw.dezyanding.ceph142.tmtpzs ceph142 *:80 running (21m) 3m ago 21m 105M - 19.2.1 f2efb0401a30 4f90102b7675
7 访问对象存储的WebUI
http://10.0.0.142/
三、s3cmd工具初始化配置
1.安装s3cmd工具包
[root@ceph141 ~]# echo 10.0.0.142 www.dezyan.com >> /etc/hosts
[root@ceph141 ~]# apt -y install s3cmd
2.创建rgw账号
[root@ceph141 ~]# radosgw-admin user create --uid "dingzhiyan" --display-name "丁志岩"
{
"user_id": "dingzhiyan",
"display_name": "丁志岩",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "dingzhiyan",
"access_key": "A3AGQ7XZLN2DL3NIR3GA",
"secret_key": "z0pbFr5riqbl40LmgzQqmLJf1aZC0xAD0KTlFkGm"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
3 运行s3cmd的运行环境，生成"/root/.s3cfg"配置文件
[root@ceph141 ~]# ll /root/.s3cfg
ls: cannot access '/root/.s3cfg': No such file or directory
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: A3AGQ7XZLN2DL3NIR3GA # rgw账号的access_key
Secret Key: z0pbFr5riqbl40LmgzQqmLJf1aZC0xAD0KTlFkGm # rgw账号的secret_key
Default Region [US]: # 直接回车即可
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: www.dezyan.com # 用于访问rgw的地址
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: www.dezyan.com/%(bucket) # 设置DNS解析风格
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: # 文件不加密，直接回车即可
Path to GPG program [/usr/bin/gpg]: # 指定自定义的gpg程序路径，直接回车即可
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No # 你的rgw是否是https，如果不是设置为No
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: # 代理服务器的地址，我并没有配置代理服务器，因此直接回车即可
New settings: # 注意，下面的信息是上面咱们填写时一个总的预览信息
Access Key: A3AGQ7XZLN2DL3NIR3GA
Secret Key: z0pbFr5riqbl40LmgzQqmLJf1aZC0xAD0KTlFkGm
Default Region: US
S3 Endpoint: www.dezyan.com
DNS-style bucket+hostname:port template for accessing a bucket: www.dezyan.com/%(bucket)
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y # 如果确认上述信息没问题的话，则输入字母Y即可。
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y # 是否保存配置，我们输入y，默认是不保存配置的。
Configuration saved to '/root/.s3cfg'
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ll /root/.s3cfg
-rw------- 1 root root 2269 Aug 23 09:59 /root/.s3cfg
四、上传测试视频案例模拟抖音，快手
1. 创建buckets
[root@ceph141 ~]# s3cmd mb s3://dezyanding-bucket
Bucket 's3://dezyanding-bucket/' created
温馨提示
通用存储桶命名规则,以下命名规则适用于通用存储桶。
- 1存储桶名称必须介于 3（最少）到 63（最多）个字符之间。
- 2.存储桶名称只能由小写字母、数字、句点（.）和连字符（-）组成。
- 3.存储桶名称必须以字母或数字开头和结尾。
- 4.存储桶名称不得包含两个相邻的句点。
- 5.存储桶名称不得采用 IP 地址格式（例如，192.168.5.4）。
- 6.存储桶名称不得以前缀 xn-- 开头。
- 7.存储桶名称不得以前缀 sthree- 开头。
- 8.存储桶名称不得以前缀 sthree-configurator 开头。
- 9.存储桶名称不得以前缀 amzn-s3-demo- 开头。
- 10.存储桶名称不得以后缀 -s3alias 结尾。此后缀是为接入点别名预留的。有关更多信息，请参阅 为您的 S3 存储桶接入点使用存储桶式别名
- 11.存储桶名称不得以后缀 --ol-s3 结尾。此后缀是为对象 Lambda 接入点别名预留的。有关更多信息，请参阅 如何为您的 S3 存储桶对象 Lambda 接入点使用存储桶式别名
- 12.存储桶名称不得以后缀 .mrap 结尾。此后缀预留用于多区域接入点名称。有关更多信息，请参阅 命名 Amazon S3 多区域接入点的规则
- 13.存储桶名称不得以后缀 --x-s3 结尾。此后缀预留用于目录存储桶。有关更多信息，请参阅 目录存储桶命名规则
- 14.存储桶名称在分区内所有 AWS 区域中的所有 AWS 账户间必须是唯一的。分区是一组区域。AWS 目前有三个分区：aws（标准区域）、aws-cn（中国区域）和 aws-us-gov（AWS GovCloud (US)）
- 15.存储桶名称不能被同一分区中的另一个 AWS 账户使用，直到存储桶被删除。
- 16.与 Amazon S3 Transfer Acceleration 一起使用的存储桶名称中不能有句点（.）。
为了获得最佳兼容性，我们建议您避免在存储桶名称中使用句点（.），但仅用于静态网站托管的存储桶除外。如果您在存储桶名称中包含句点，则无法通过 HTTPS 使用虚拟主机式寻址，除非您执行自己的证书验证。这是因为用于存储桶虚拟托管的安全证书不适用于名称中带有句点的存储桶
此限制不会影响用于静态网站托管的存储桶，因为静态网站托管只能通过 HTTP 提供。有关虚拟主机式寻址的更多信息，请参阅存储桶的虚拟托管。有关静态网站托管的更多信息，请参阅使用 Amazon S3 托管静态网站。
参考链接:
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/bucketnamingrules.html
2.查看buckets
[root@ceph141 ~]# s3cmd ls
2025-04-03 04:01 s3://dezyanding-bucket
[root@ceph141 ~]# radosgw-admin buckets list
[
"dezyanding-bucket"
]
3.使用s3cmd上传数据到buckets
[root@ceph141 /]# s3cmd put 01-昨日内容回顾及今日内容预告.mp4 s3://dezyanding-bucket
s3://dezyanding-bucket
upload: '01-昨日内容回顾及今日内容预告.mp4' -> 's3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4' [part 1 of 4, 15MB] [1 of 1]
15728640 of 15728640 100% in 3s 4.76 MB/s done
upload: '01-昨日内容回顾及今日内容预告.mp4' -> 's3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4' [part 2 of 4, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 62.77 MB/s done
upload: '01-昨日内容回顾及今日内容预告.mp4' -> 's3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4' [part 3 of 4, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 17.88 MB/s done
upload: '01-昨日内容回顾及今日内容预告.mp4' -> 's3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4' [part 4 of 4, 8MB] [1 of 1]
8787432 of 8787432 100% in 0s 51.04 MB/s done
[root@ceph141 /]# echo 15728640/1024/1024 | bc
15
[root@ceph141 /]# s3cmd put 02-对象存储网关组件rgw概述.mp4 s3://dezyanding-bucket
………………
[root@ceph141 /]# s3cmd ls s3://dezyanding-bucket
2025-04-03 04:04 55973352 s3://dezyanding-bucket/01-老男孩教育-昨日内容回顾及今日内容预告.mp4
2025-04-03 04:05 168659396 s3://dezyanding-bucket/02-老男孩教育-对象存储网关组件rgw概述.mp4
4.使用s3cmd下载数据
[root@ceph141 /]# s3cmd get s3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4 /tmp/
download: 's3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4' -> '/tmp/01-昨日内容回顾及今日内容预告.mp4' [1 of 1]
55973352 of 55973352 100% in 0s 108.94 MB/s done
5.授权策略
参考链接:
https://docs.ceph.com/en/squid/radosgw/bucketpolicy/
[root@ceph141 ~]# cat dezyanding-anonymous-access-policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::dezyanding-bucket/*"
]
}]
}
6.添加授权策略
[root@ceph141 /]# s3cmd info s3://dezyanding-bucket
s3://dezyanding-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 丁志岩: FULL_CONTROL
[root@ceph141 /]# s3cmd info s3://dezyanding-bucket
s3://dezyanding-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::dezyanding-bucket/*"
]
}]
}
CORS: none
ACL: 丁志岩: FULL_CONTROL
7.访问测试
http://10.0.0.142/dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4
http://10.0.0.142/dezyanding-bucket/02-对象存储网关组件rgw概述.mp4
8.删除策略
[root@ceph141 ~]# s3cmd info s3://dezyanding-bucket
s3://dezyanding-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::dezyanding-bucket/*"
]
}]
}
CORS: none
ACL: 丁志岩: FULL_CONTROL
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd delpolicy s3://dezyanding-bucket
s3://dezyanding-bucket/: Policy deleted
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd info s3://dezyanding-bucket
s3://dezyanding-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 丁志岩: FULL_CONTROL
9.再次访问测试（无权限）
http://10.0.0.142/dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4
http://10.0.0.142/dezyanding-bucket/02-对象存储网关组件rgw概述.mp4
五、s3cmd基础操作实战
推荐阅读:
https://www.s3express.com/help/help.html
1.查看buckets的列表
[root@ceph141 ~]# s3cmd ls
2025-04-03 04:01 s3://dezyanding-bucket
2.查看buckets的数据
[root@ceph141 ~]# s3cmd la
2025-04-03 04:04 55973352 s3://dezyanding-bucket/01-昨日内容回顾及今日内容预告.mp4
2025-04-03 04:05 168659396 s3://dezyanding-bucket/02-对象存储网关组件rgw概述.mp4
3.上传多个文件到buckets
[root@ceph141 ~]# s3cmd put /etc/os-release /etc/fstab /etc/passwd s3://dezyanding-bucket/dezyan/
4.下载文件
[root@ceph141 ~]# s3cmd get s3://dezyanding-bucket/dezyan/fstab /tmp/xixi
download: 's3://dezyanding-bucket/dezyan/fstab' -> '/tmp/xixi' [1 of 1]
657 of 657 100% in 0s 13.38 KB/s done
[root@ceph141 ~]#
[root@ceph141 ~]# diff /tmp/xixi /etc/fstab
5.删除对象
[root@ceph141 ~]# s3cmd del s3://dezyanding-bucket/dezyan/fstab
delete: 's3://dezyanding-bucket/dezyan/fstab'
[root@ceph141 ~]# s3cmd rm s3://dezyanding-bucket/dezyan/passwd # rm是del的别名
delete: 's3://dezyanding-bucket/dezyan/passwd'
6.将目录的所有文件上川到bucket
[root@ceph141 ~]# s3cmd sync /dezyan/softwares/docker/ s3://dezyanding-bucket
7.查看某个bucket的大小
[root@ceph141 ~]# s3cmd du s3://dezyanding-bucket
485923394 12 objects s3://dezyanding-bucket/
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd du s3://dezyanding-bucket -H
463M 12 objects s3://dezyanding-bucket/
8.获取bucket或者objects信息
[root@ceph141 ~]# s3cmd info s3://dezyanding-bucket
s3://dezyanding-bucket/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 丁志岩: FULL_CONTROL
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd info s3://dezyanding-bucket/runc
s3://dezyanding-bucket/runc (object):
File size: 14214624
Last mod: Thu, 03 Apr 2025 06:43:41 GMT
MIME type: application/x-executable
Storage: STANDARD
MD5 sum: 1b74d0525e0a805845473b7207446757
SSE: none
Policy: none
CORS: none
ACL: 丁志岩: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1743579172/ctime:1743405727/gid:1000/gname:dezyanding/md5:1b74d0525e0a805845473b7207446757/mode:33261/mtime:1680632676/uid:1000/uname:dezyanding
9.拷贝数据
[root@ceph141 ~]# s3cmd mb s3://dezyan-linux96
Bucket 's3://dezyan-linux96/' created
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd ls s3://dezyan-linux96
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd cp s3://dezyanding-bucket/runc s3://dezyan-linux96
remote copy: 's3://dezyanding-bucket/runc' -> 's3://dezyan-linux96/runc' [1 of 1]
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd ls s3://dezyan-linux96
2025-04-03 06:47 14214624 s3://dezyan-linux96/runc
10.修改对象的元数据
[root@ceph141 ~]# s3cmd ls s3://dezyan-linux96
2025-04-03 06:47 14214624 s3://dezyan-linux96/runc
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd modify s3://dezyan-linux96/runc
modify: 's3://dezyan-linux96/runc' [1 of 1]
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd ls s3://dezyan-linux96/
2025-04-03 06:48 14214624 s3://dezyan-linux96/runc
#只修改了时间？？？
11.移动objects对象
[root@ceph141 ~]# s3cmd mv s3://dezyanding-bucket/containerd s3://dezyan-linux96/
move: 's3://dezyanding-bucket/containerd' -> 's3://dezyan-linux96/containerd' [1 of 1]
12.查看正在上传的文件列表
[root@ceph141 ~]# s3cmd put dezyan-grafana-v10.4.0.tar.gz s3://dezyan-linux96
# 单独用一个终端测试
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd multipart s3://dezyan-linux96
s3://dezyan-linux96/
Initiated Path Id
2025-04-03T06:52:30.386Z s3://dezyan-linux96/dezyan-grafana-v10.4.0.tar.gz 2~vknzQkX2QnqFX8p4KWRDrcfAq3VM1El
13.递归删除bucket
[root@ceph141 /]# s3cmd rb s3://dezyanding-bucket -r
六、buckets权限策略
参考链接:
https://docs.ceph.com/en/squid/radosgw/bucketpolicy/
s3:PutObjectAcl
上传对象。
s3:GetObject
下载对象。
s3:CreateBucket
创建buckets。
s3:ListBucket
查看buckets。
s3:DeleteObject
删除对象。
s3:DeleteBucket
删除buckets。
七、Python操作对象存储服务实战
参考链接:
https://docs.ceph.com/en/squid/radosgw/s3/python/
1.安装python环境
1.1 安装pip工具包
[root@elk93 ~]# apt -y install python3-pip
1.2 安装boto包
[root@elk93 ~]# pip install boto==2.49.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
2.编写python程序
[root@elk93 ~]# cat rgw-dezyanding.py
import boto
import boto.s3.connection
# 你账号的access_key和secret_key需要自行修改
access_key = 'A3AGQ7XZLN2DL3NIR3GA'
secret_key = 'z0pbFr5riqbl40LmgzQqmLJf1aZC0xAD0KTlFkGm'
# 连接rgw
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = '10.0.0.142',
is_secure=False,
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
# 创建bucket
bucket = conn.create_bucket('dezyanding-rgw')
# 查看bucket列表
for bucket in conn.get_all_buckets():
print("{name}\t{created}".format(
name = bucket.name,
created = bucket.creation_date,
))
# 查看bucket内容
for key in bucket.list():
print("{name}\t{size}\t{modified}".format(
name = key.name,
size = key.size,
modified = key.last_modified,
))
# 创建一个对象
key = bucket.new_key('blog.txt')
key.set_contents_from_string('https://www.cnblogs.com/dezyanding\n')
# 生成对象下载的URL
hello_key = bucket.get_key('blog.txt')
hello_url = hello_key.generate_url(0, query_auth=False, force_http=True)
print(hello_url)
3.运行程序
[root@elk93 ~]# python3 rgw-dezyanding.py
dezyan-linux96 2025-04-03T06:47:20.497Z
dezyanding-rgw 2025-04-03T07:21:15.232Z
blog.txt 36 2025-04-03T07:22:50.424Z
http://10.0.0.142/dezyanding-rgw/blog.txt
4.访问测试，发现无法访问！
[root@elk93 ~]# curl http://10.0.0.142/dezyanding-rgw/blog.txt;echo
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message></Message><BucketName>dezyanding-rgw</BucketName><RequestId>tx0000090f801ec50aef41b-0067ee3899-64531-default</RequestId><HostId>64531-default-default</HostId></Error>
[root@ceph141 ~]# python3 rgw-dezyanding.py # 再次测试发现输出内容和之前有变化，原因是已经有bucket。
dezyanding-bucket 2024-12-06T04:14:11.346Z
dezyanding-rgw 2024-12-06T07:55:59.434Z
blog.txt 35 2024-12-06T07:55:59.553Z
http://www.dezyanding.com/dezyanding-rgw/blog.txt
4.使用s3cmd命令访问测试
[root@ceph141 /]# s3cmd get s3://dezyanding-rgw/blog.txt
download: 's3://dezyanding-rgw/blog.txt' -> './blog.txt' [1 of 1]
36 of 36 100% in 0s 847.12 B/s done
[root@ceph141 /]#
[root@ceph141 /]# cat blog.txt
https://www.cnblogs.com/dezyanding
[root@ceph141 /]#
5.创建访问策略
5.1 编写策略配置文件
[root@ceph141 ~]# cat dezyanding-anonymous-access-policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::dezyanding-rgw/*"
]
}]
}
5.2 应用策略
[root@ceph141 ~]# s3cmd info s3://dezyanding-rgw
s3://dezyanding-rgw/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: none
CORS: none
ACL: 丁志岩: FULL_CONTROL
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd setpolicy dezyanding-anonymous-access-policy.json s3://dezyanding-rgw
s3://dezyanding-rgw/: Policy updated
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd info s3://dezyanding-rgw
s3://dezyanding-rgw/ (bucket):
Location: default
Payer: BucketOwner
Expiration Rule: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": ["*"]},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::dezyanding-rgw/*"
]
}]
}
CORS: none
ACL: 丁志岩: FULL_CONTROL
6.再次访问测试
[root@elk93 ~]# curl http://10.0.0.142/dezyanding-rgw/blog.txt
八、对于swift工具操作对象存储网关rgw
1.Switft API接口概述
1.1 什么是swift
swift的用户账号对应radosgw中的subuser(子用户)，它隶属于某个事先存在的user(用户账号)。
Swift API的上下文中，存储桶以container表示，而非S3中的bucket，但二者在功能上相似，都是对象数据的容器。
Python swiftclient是一个用于和swift API交互的python客户端程序，它包含了Python API(swift模块)和一个swift命令。
swift命令可以通过Swift API完成容器和对象数据的管理操作。
1.2 swift实现的基本逻辑
- 1.创建专属的用户名和子用户授权;
- 2.安装专属的客户端命令和python模块;
- 3.配置专属的认证配置文件;
- 4.综合测试swift的资源对象管理;
2.swift命令行配置实战
2.1 创建账号
[root@ceph141 ~]# radosgw-admin user create --uid "dezyanding" --display-name "丁志岩"
{
"user_id": "dezyanding",
"display_name": "丁志岩",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "dezyanding",
"access_key": "030NCR3HK8G1GSJ2N7OG",
"secret_key": "4v2h8asPwUZhS7BGS6pWwmUpRX9l44veiygl7tmG"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
2.2 查看账号列表
[root@ceph141 ~]# radosgw-admin user list
[
"dashboard",
"dezyanding",
"dingzhiyan"
]
2.3 基于现有用户创建子用户
[root@ceph141 ~]# radosgw-admin subuser create --uid=dezyanding --subuser=dezyanding:swift --access=full
{
"user_id": "dezyanding",
"display_name": "丁志岩",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [
{
"id": "dezyanding:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "dezyanding",
"access_key": "0DEMUF5H9Q8PQKLA4S4S",
"secret_key": "Q1s01Ax5r7nof1mNCHwG3PRsKs5HAs9wSaVxpfAQ"
}
],
"swift_keys": [
{
"user": "dezyanding:swift",
"secret_key": "Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD"
}
],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
2.5 安装工具包
[root@ceph141 ~]# apt -y install python-setuptools python3-pip
[root@ceph141 ~]# pip install python-swiftclient==4.6.0
2.6 创建并查看存储桶
[root@ceph141 ~]# swift -A http://10.0.0.142/auth -U dezyanding:swift -K Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD post dezyanding-swift
[root@ceph141 ~]#
[root@ceph141 ~]# swift -A http://10.0.0.142/auth -U dezyanding:swift -K Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD list
dezyanding-swift
[root@ceph141 ~]#
[root@ceph141 ~]# s3cmd ls
2025-04-03 06:47 s3://dezyan-linux96
2025-04-03 07:21 s3://dezyanding-rgw
温馨提示
从结果上来看，貌似s3cmd和swift二者并不兼容，比如存储桶的和管理方式都不一样
相关参数说明
-A
指定认证的URL
-U
指定子用户的名称
-K
指定KEY信息
2.7 上传文件到存储桶
[root@ceph141 ~]# swift -A http://10.0.0.142/auth -U dezyanding:swift -K Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD upload dezyanding-swift /etc/os-release
etc/os-release
[root@ceph141 ~]#
[root@ceph141 ~]# swift -A http://10.0.0.142/auth -U dezyanding:swift -K Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD upload dezyanding-swift /etc/fstab
etc/fstab
[root@ceph141 ~]#
[root@ceph141 ~]# swift -A http://10.0.0.142/auth -U dezyanding:swift -K Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD list dezyanding-swift # 很明显数据上传成功
etc/fstab
etc/os-release
2.8 配置环境变量
[root@ceph141 ~]# cat .swift
export ST_AUTH=http://10.0.0.142/auth
export ST_USER=dezyanding:swift
export ST_KEY=Mhq5BWRJAVXLbzFNBFOXC9XlS3FiGtHkYEYDhJSD
[root@ceph141 ~]#
[root@ceph141 ~]# source .swift
[root@ceph141 ~]#
[root@ceph141 ~]# swift list dezyanding-swift
etc/fstab
etc/os-release
九、ceph集群的维护命令
1.查看ceph集群的服务
[root@ceph141 ~]# ceph orch ls
2.查看ceph集群的守护进程
[root@ceph141 ~]# ceph orch ps
[root@ceph141 ~]# ceph orch ls rgw --export
[root@ceph141 ~]# ceph orch ps --daemon_type=rgw
[root@ceph141 ~]# ceph orch ps --daemon_type=osd
[root@ceph141 ~]# ceph orch ps --daemon_type=prometheus
[root@ceph141 ~]# ceph orch ps --service_name=rgw.dezyanding
3.查看指定节点的守护进程
[root@ceph141 ~]# ceph orch ps ceph141
4.重启指定节点守护进程服务
[root@ceph141 ~]# ceph orch daemon restart node-exporter.ceph143
5.查看主机有哪些设备列表
[root@ceph141 ~]# ceph orch device ls
6.查看集群有哪些主机列表
[root@ceph141 ~]# ceph orch host ls
7.报告配置的后端及其状态
[root@ceph141 ~]# ceph orch status
Backend: cephadm
Available: Yes
Paused: No
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch status --detail
Backend: cephadm
Available: Yes
Paused: No
Host Parallelism: 10
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch status --detail --format json
{"available": true, "backend": "cephadm", "paused": false, "workers": 10}
[root@ceph141 ~]
8.检查服务版本与可用和目标容器
[root@ceph141 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/ceph/ceph v19 f2efb0401a30 8 weeks ago 1.3GB
quay.io/prometheus/prometheus v2.51.0 1d3b7f56885b 12 months ago 262MB
quay.io/ceph/grafana 10.4.0 c8b91775d855 13 months ago 430MB
quay.io/prometheus/node-exporter v1.7.0 72c9c2088986 16 months ago 22.7MB
quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 2 years ago 65.1MB
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch upgrade check quay.io/ceph/ceph:v19
{
"needs_update": {},
"non_ceph_image_daemons": [
"prometheus.ceph141",
"grafana.ceph141",
"node-exporter.ceph141",
"alertmanager.ceph141",
"node-exporter.ceph142",
"node-exporter.ceph143"
],
"target_digest": "quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de",
"target_id": "f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d",
"target_name": "quay.io/ceph/ceph:v19",
"target_version": "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)",
"up_to_date": [
"osd.1",
"crash.ceph141",
"osd.2",
"mgr.ceph141.mbakds",
"rgw.dezyanding.ceph141.svisis",
"osd.0",
"mon.ceph141",
"mds.dezyan-cephfs.ceph141.pthitg",
"ceph-exporter.ceph141",
"osd.4",
"ceph-exporter.ceph142",
"rgw.dezyanding.ceph142.tmtpzs",
"mon.ceph142",
"mgr.ceph142.qgifwo",
"crash.ceph142",
"mds.dezyan-cephfs.ceph142.pmzglk",
"osd.3",
"osd.5",
"osd.6",
"osd.8",
"osd.7",
"mon.ceph143",
"ceph-exporter.ceph143",
"crash.ceph143"
]
}
十、ceph集群的监控及常用指标概述
1.查看集群的架构
[root@ceph141 ~]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 9m ago 3d count:1
ceph-exporter 3/3 9m ago 3d *
crash 3/3 9m ago 3d *
grafana ?:3000 1/1 9m ago 3d count:1
mds.dezyan-cephfs 2/2 9m ago 29h count:2
mgr 2/2 9m ago 3d count:2
mon 3/5 9m ago 3d count:5
node-exporter ?:9100 3/3 9m ago 3d *
osd 9 9m ago - <unmanaged>
prometheus ?:9095 1/1 9m ago 3d count:1
rgw.dezyanding ?:80 2/2 9m ago 5h count:2
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --service_name alertmanager
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
alertmanager.ceph141 ceph141 *:9093,9094 running (2d) 9m ago 3d 30.0M - 0.25.0 c8568f914cd2 4b8bc4b1fc5b
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --service_name grafana
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
grafana.ceph141 ceph141 *:3000 running (2d) 9m ago 3d 155M - 10.4.0 c8b91775d855 e5a398893001
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --service_name node-exporter
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
node-exporter.ceph141 ceph141 *:9100 running (2d) 10m ago 3d 19.0M - 1.7.0 72c9c2088986 cd20eec4cf53
node-exporter.ceph142 ceph142 *:9100 running (26h) 10m ago 3d 18.9M - 1.7.0 72c9c2088986 08ef8871f112
node-exporter.ceph143 ceph143 *:9100 running (5m) 5m ago 2d 2811k - 1.7.0 72c9c2088986 3a895f87bad2
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --service_name prometheus
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
prometheus.ceph141 ceph141 *:9095 running (2d) 10m ago 3d 121M - 2.51.0 1d3b7f56885b abeb2ed5eab2
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch ps --service_name ceph-exporter
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
ceph-exporter.ceph141 ceph141 running (2d) 10m ago 3d 15.7M - 19.2.1 f2efb0401a30 0cac3a5b6be7
ceph-exporter.ceph142 ceph142 running (26h) 10m ago 3d 8103k - 19.2.1 f2efb0401a30 f8d8762f14d0
ceph-exporter.ceph143 ceph143 running (2d) 5m ago 2d 23.2M - 19.2.1 f2efb0401a30 e85d535ca925
[root@ceph141 ~]#
2.访问prometheus的WebUI
http://10.0.0.141:9095/targets
3.访问Alertmanager的WebUI
http://10.0.0.141:9093/#/alerts
4.访问grafana的WebUI
https://10.0.0.141:3000/
5.常用指标概述
参考链接:
https://docs.ceph.com/en/squid/monitoring/#ceph-metrics
https://docs.ceph.com/en/squid/mgr/dashboard/
十一、CRUSH初体验将数据写入不同的OSD设备
1.CRUSH Map概述
CRUSH英文全称为"Controlled Replication Under Scalable Hashing"，是Ceph的核心设计之一，它本质上是ceph存储集群使用的一种数据分发算法，类似于openstack的swift的AQS对象存储所使用的哈希和一致性hash数据分布算法。
CRUSH算法通过接收多维参数，通过一定的计算对客户端对象数据进行分布存位置的确定，来解决数据动态分发的问题。因此ceph客户端无需经过传统查表的方式来获取数据的索引，进而根据索引来读取数据，只需通过crush算法计算后直接和对应的OSD交互进行数据读写。这样，ceph就避免了查表这种传统中心化存在的单点故障，性能瓶颈以及不易扩展的缺陷。这就是ceph相较于其他分布式存储系统具有高扩展性，高可用和高性能特点的主要原因。
ceph中的寻找至少要经历以下三次映射:
- 1.File和object映射:
文件数据object的数据块切片操作，便于多数据的并行化处理。
- 2.Object和PG映射:
将文件数据切分后的每一个Object通过简单的Hash算法归到一个PG中。
- 3.PG和OSD映射:
将PG映射到主机实际的OSD数据磁盘上。
CRUSH算法提供了配置和更改和数据动态再平衡等关键特性，而CRUSH算法存储数据对象的过程可通过CRUSH Map控制并进行自定义修改，CRUSH map是ceph集群物理拓扑结构，副本策略以及故障域等信息抽象配置段，借助于CRUSH Map可以将数据伪随机地分布到集群的各个OSD上。
OSD出现异常的时候，为了避免故障风暴，往往会实现一个所谓的故障域。
2.环境准备
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 5.34389 root default
-3 1.78130 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 1.78130 host ceph143
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
[root@ceph141 ~]#
3.从monitor节点上获取CRUSH map
[root@ceph141 ~]# ceph osd getcrushmap -o dezyanding-hdd.file
25
[root@ceph141 ~]#
[root@ceph141 ~]# file dezyanding-hdd.file
dezyanding-hdd.file: data
[root@ceph141 ~]#
[root@ceph141 ~]# ll dezyanding-hdd.file
-rw-r--r-- 1 root root 1173 Apr 3 17:35 dezyanding-hdd.file
4.获取该crushmap文件后，编译为可读文件
[root@ceph141 ~]# apt -y install ceph-base
[root@ceph141 ~]#
[root@ceph141 ~]# crushtool -d dezyanding-hdd.file -o dezyanding-hdd-ssd.file
[root@ceph141 ~]#
[root@ceph141 ~]# file dezyanding-hdd*
dezyanding-hdd.file: data
dezyanding-hdd-ssd.file: ASCII text
5.查看修改前的内容
[root@ceph141 ~]# cat dezyanding-hdd-ssd.file
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root
# buckets
host ceph141 {
id -3 # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
# weight 1.78130
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.29300
item osd.1 weight 0.48830
item osd.2 weight 1.00000
}
host ceph142 {
id -5 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
# weight 1.78130
alg straw2
hash 0 # rjenkins1
item osd.3 weight 0.29300
item osd.4 weight 0.48830
item osd.5 weight 1.00000
}
host ceph143 {
id -7 # do not change unnecessarily
id -8 class hdd # do not change unnecessarily
# weight 1.78130
alg straw2
hash 0 # rjenkins1
item osd.6 weight 0.29300
item osd.7 weight 0.48830
item osd.8 weight 1.00000
}
root default {
id -1 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 5.34389
alg straw2
hash 0 # rjenkins1
item ceph141 weight 1.78130
item ceph142 weight 1.78130
item ceph143 weight 1.78130
}
# rules
rule replicated_rule {
id 0
type replicated
step take default
step chooseleaf firstn 0 type host
step emit
}
rule erasure-code {
id 1
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default
step chooseleaf indep 0 type host
step emit
}
# end crush map
6.修改文本文件内容
[root@ceph141 ~]# cat dezyanding-hdd-ssd.file
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root
# buckets
host ceph141 {
id -3 # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
# weight 1.78130
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.29300
item osd.1 weight 0.48830
item osd.2 weight 1.00000
}
host ceph142 {
id -5 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
# weight 1.78130
alg straw2
hash 0 # rjenkins1
item osd.3 weight 0.29300
item osd.4 weight 0.48830
item osd.5 weight 1.00000
}
host ceph143 {
id -7 # do not change unnecessarily
id -8 class hdd # do not change unnecessarily
# weight 1.78130
alg straw2
hash 0 # rjenkins1
item osd.6 weight 0.29300
item osd.7 weight 0.48830
item osd.8 weight 1.00000
}
host ceph141-hdd {
id -13 # do not change unnecessarily
id -14 class hdd # do not change unnecessarily
# weight 1.75789
alg straw2
hash 0 # rjenkins1
item osd.2 weight 0.97659
}
host ceph142-hdd {
id -15 # do not change unnecessarily
id -16 class hdd # do not change unnecessarily
# weight 2.75789
alg straw2
hash 0 # rjenkins1
item osd.5 weight 0.97659
}
host ceph143-hdd {
id -17 # do not change unnecessarily
id -18 class hdd # do not change unnecessarily
# weight 1.75789
alg straw2
hash 0 # rjenkins1
item osd.8 weight 0.97659
}
root default {
id -1 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 5.34389
alg straw2
hash 0 # rjenkins1
item ceph141 weight 1.78130
item ceph142 weight 1.78130
item ceph143 weight 1.78130
}
root ceph-hdd {
id -11 # do not change unnecessarily
id -12 class hdd # do not change unnecessarily
# weight 6.27367
alg straw2
hash 0 # rjenkins1
item ceph141-hdd weight 1.75789
item ceph142-hdd weight 2.75789
item ceph143-hdd weight 1.75789
}
# rules
rule replicated_rule {
id 0
type replicated
step take default
step chooseleaf firstn 0 type host
step emit
}
rule erasure-code {
id 1
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default
step chooseleaf indep 0 type host
step emit
}
rule dezyan_hdd_rule {
id 10
type replicated
step take ceph-hdd
step chooseleaf firstn 0 type host
step emit
}
# end crush map
7.应用配置文件
[root@ceph141 ~]# crushtool -c dezyanding-hdd-ssd.file -o dezyanding-hdd-ssd.crushmap
[root@ceph141 ~]#
[root@ceph141 ~]# file dezyanding-hdd-ssd.crushmap
dezyanding-hdd-ssd.crushmap: data
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd setcrushmap -i dezyanding-hdd-ssd.crushmap
26
8.查看OSD信息
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-11 6.27367 root ceph-hdd
-13 1.75789 host ceph141-hdd
2 hdd 0.97658 osd.2 up 1.00000 1.00000
-15 2.75789 host ceph142-hdd
5 hdd 0.97658 osd.5 up 1.00000 1.00000
-17 1.75789 host ceph143-hdd
8 hdd 0.97658 osd.8 up 1.00000 1.00000
-1 5.34389 root default
-3 1.78130 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 1.78130 host ceph143
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
9.创建存储池测试自定义规则
9.1.创建存储池
[root@ceph141 ~]# ceph osd pool create dezyan-hdd 8 8 replicated dezyan_hdd_rule
pool 'dezyan-hdd' created
9.2.查看存储池对应的规则id（启动ssd对应的crush_rule为1，而hdd对应的crush_rule为2）
[root@ceph141 ~]# ceph osd pool ls detail | grep hdd
pool 21 'dezyan-hdd' replicated size 3 min_size 2 crush_rule 10 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 1270 flags hashpspool stripe_width 0 read_balance_score 1.50
9.3.查看ssd的PG底层对应的OSD范围为[7-9，共计3块磁盘]
[root@ceph141 ~]# ceph pg ls-by-pool dezyan-hdd
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING
21.0 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [5,2,8]p5 [5,2,8]p5 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T19:04:34.938020+0000
21.1 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [2,5,8]p2 [2,5,8]p2 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T12:34:40.325695+0000
21.2 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [5,8,2]p5 [5,8,2]p5 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T16:51:12.557603+0000
21.3 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [5,2,8]p5 [5,2,8]p5 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T21:22:02.905517+0000
21.4 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [2,8,5]p2 [2,8,5]p2 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T14:12:50.656110+0000
21.5 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [5,8,2]p5 [5,8,2]p5 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T11:01:59.247146+0000
21.6 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [8,5,2]p8 [8,5,2]p8 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T16:40:00.911357+0000
21.7 0 0 0 0 0 0 0 0 0 active+clean 43s 0'0 1270:13 [2,5,8]p2 [2,5,8]p2 2025-04-03T09:50:24.897815+0000 2025-04-03T09:50:24.897815+0000 0 periodic scrub scheduled @ 2025-04-04T10:00:09.178814+0000
* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details.
10.测试验证
10.1 写入测试数据
[root@ceph141 ~]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 5.3 TiB 5.3 TiB 90 GiB 90 GiB 1.65
TOTAL 5.3 TiB 5.3 TiB 90 GiB 90 GiB 1.65
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
...
dezyan-hdd 21 8 0 B 0 0 B 0 957 GiB
[root@ceph141 ~]#
[root@ceph141 ~]# ll -h /etc/hosts
-rw-r--r-- 1 root root 312 Apr 3 11:50 /etc/hosts
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# rados put myhosts /etc/hosts -p dezyan-hdd
[root@ceph141 ~]#
[root@ceph141 ~]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 5.3 TiB 5.3 TiB 90 GiB 90 GiB 1.65
TOTAL 5.3 TiB 5.3 TiB 90 GiB 90 GiB 1.65
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
...
dezyan-hdd 21 8 312 B 1 12 KiB 0 957 GiB
10.2 查看对象对应的OSD映射关系
[root@ceph141 ~]# ceph osd map dezyan-hdd myhosts
osdmap e1270 pool 'dezyan-hdd' (21) object 'myhosts' -> pg 21.5a39779b (21.3) -> up ([5,2,8], p5) acting ([5,2,8], p5)
推荐阅读:
https://www.cnblogs.com/dezyanding/p/18403765
ceph集群的OSD缩容
参考链接:
https://www.cnblogs.com/dezyanding/p/18370804
本文来自博客园，作者：丁志岩，转载请注明原文链接：https://www.cnblogs.com/dezyan/p/18811450
浙公网安备 33010602011771号