Ceph radosgw的基本使用

 RadosGW 对象存储网关简介

  RadosGW 是对象存储(OSS,Object Storage Service)的一种访问实现方式，RADOS 网关也称为 Ceph 对象网关、RadosGW、RGW，是一种服务，使客户端能够利用标准对象存储API 来访问 Ceph 集群，它支持 AWS S3 和 Swift API，在 ceph 0.8 版本之后使用 Civetweb(https://github.com/civetweb/civetweb) 的 web 服务器来响应 api 请求，客户端使用http/https 协议通过 RESTful API 与 RGW 通信，而 RGW 则通过 librados 与 ceph 集群通信，RGW 客户端通过 s3 或者 swift api 使用 RGW 用户进行身份验证，然后 RGW 网关代表用户利用 cephx 与 ceph 存储进行身份验证。

  S3 由 Amazon 于 2006 年推出，全称为 Simple Storage Service,S3 定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3 就是对象存储，对象存储就是 S3,它是对象存储市场的霸主，后续的对象存储都是对 S3 的模仿。

 

回到顶部

  部署 RadosGW 服务：

  将 ceph-mgr1、ceph-mgr2 服务器部署为高可用的 radosGW 服务

  添加ceph仓库源并安装radosgw

	#支持 https 镜像仓库源：
	apt install -y apt-transport-https ca-certificates curl software-properties-common
	#导入 key：
	wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add -
	apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific/ bionic main'
	root@ceph-mgr1:/etc/apt# apt update
	root@ceph-mgr1:~# apt-cache madison radosgw #搜索radosgw
	radosgw | 16.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages
	radosgw | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 Packages
	radosgw | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security/main amd64 Packages
	radosgw | 12.2.4-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/main amd64 Packages

 

	#mg1和mgr2安装radosgw
	root@ceph-mgr1:/etc/apt# apt install radosgw
	root@ceph-mgr1:~# radosgw -v
	ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
	root@ceph-mgr2:~# radosgw -v
	ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

 

   ceph-deploy节点调度 mgr1 和mg2 部署rgw服务

	root@ceph-deploy:~# su - cephadmin
	cephadmin@ceph-deploy:~$ cd ceph-cluster/
	cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr1
	cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr2

  --overwrite-conf 参数含义：以当前ceph-deploy的ceph.conf配置为准，替换掉mgr1节点上的/etc/ceph.conf

  

  RGW部署完成后，会在mgr1、mgr2节点启动ceph-radosgw@rgw.ceph-mgr1、2服务，端口监听在tcp 7480

 

 

 

   mgr1节点验证radosgw服务

	#检查radosgw服务
	root@ceph-mgr1:~# systemctl status ceph-radosgw@rgw.ceph-mgr1.service
	root@ceph-mgr2:~# systemctl status ceph-radosgw@rgw.ceph-mgr2.service
	#检查服务端口
	root@ceph-mgr1:~# ss -lntup|grep 7480
	root@ceph-mgr2:~# ss -lntup|grep 7480

 

 

  使用web http方式访问mgr1、mgr2 ip:7480服务

 

 

  从ceph状态验证查看rgw服务部署

 

回到顶部

  Radosgw默认存储池

  初始化完成 radosgw 之后，会初始化默认的存储池如下：

  名称以 default.rgw.* 为前缀和 .rgw.root的存储池

	cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
	device_health_metrics
	rbd-data
	default.rgw.log
	.rgw.root
	default.rgw.control
	default.rgw.meta
	cephfs-metadata
	cephfs-data

 

回到顶部

  验证radosgw服务进程

	root@ceph-mgr1:~# ps -ef|grep radosgw
	ceph 1302 1 0 14:58 ? 00:00:04 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph
	root 3562 3492 0 15:32 pts/0 00:00:00 grep --color=auto radosgw
	root@ceph-mgr2:~# ps -ef|grep radosgw
	ceph 19646 1 0 15:22 ? 00:00:01 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr2 --setuser ceph --setgroup ceph
	root 20332 2930 0 15:33 pts/0 00:00:00 grep --color=auto radosgw

 

回到顶部

  radosgw 的存储池类型：

	cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
	device_health_metrics
	rbd-data
	default.rgw.log
	.rgw.root
	default.rgw.control
	default.rgw.meta
	cephfs-metadata
	cephfs-data

 

  查看默认 radosgw 的存储池信息：

	cephadmin@ceph-deploy:~/ceph-cluster$ radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default
	{
	"id": "638985bc-6486-4a1a-8012-a619266611ef",
	"name": "default",
	"domain_root": "default.rgw.meta:root",
	"control_pool": "default.rgw.control",
	"gc_pool": "default.rgw.log:gc",
	"lc_pool": "default.rgw.log:lc",
	"log_pool": "default.rgw.log",
	"intent_log_pool": "default.rgw.log:intent",
	"usage_log_pool": "default.rgw.log:usage",
	"roles_pool": "default.rgw.meta:roles",
	"reshard_pool": "default.rgw.log:reshard",
	"user_keys_pool": "default.rgw.meta:users.keys",
	"user_email_pool": "default.rgw.meta:users.email",
	"user_swift_pool": "default.rgw.meta:users.swift",
	"user_uid_pool": "default.rgw.meta:users.uid",
	"otp_pool": "default.rgw.otp",
	"system_key": {
	"access_key": "",
	"secret_key": ""
	},
	"placement_pools": [
	{
	"key": "default-placement",
	"val": {
	"index_pool": "default.rgw.buckets.index",
	"storage_classes": {
	"STANDARD": {
	"data_pool": "default.rgw.buckets.data"
	}
	},
	"data_extra_pool": "default.rgw.buckets.non-ec",
	"index_type": 0
	}
	}
	],
	"realm_id": "",
	"notif_pool": "default.rgw.log:notif"
	}

  rgw.root： 包含 realm(领域信息)，比如 zone 和 zonegroup

  default.rgw.log： 存储日志信息，用于记录各种 log 信息。

  default.rgw.control： 系统控制池，在有数据更新时，通知其它 RGW 更新缓存。

  default.rgw.meta： 元数据存储池，通过不同的名称空间分别存储不同的 rados 对象，这些名称空间包括⽤⼾UID 及其 bucket 映射信息的名称空间 users.uid、⽤⼾的密钥名称空间users.keys、⽤⼾的 email 名称空间 users.email、⽤⼾的 subuser 的名称空间 users.swift，以及 bucket 的名称空间 root 等。

  default.rgw.buckets.index： 存放 bucket 到 object 的索引信息。

  default.rgw.buckets.data： 存放对象的数据。

  default.rgw.buckets.non-ec： 数据的额外信息存储池

  default.rgw.users.uid： 存放用户信息的存储池。

  default.rgw.data.root： 存放 bucket 的元数据，结构体对应 RGWBucketInfo，比如存放桶名、桶 ID、data_pool 等。

 

  查看对象存储池的存储策略、副本数量、pgp和pg的数量

	cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta crush_rule
	crush_rule: replicated_rule
	cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta size
	size: 3
	cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta pgp_num
	pgp_num: 8
	cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool get default.rgw.meta pg_num
	pg_num: 8

 

回到顶部

  radosgw http 服务高可用配置

  自定义 http 端口

  配置文件可以在 ceph deploy 服务器修改然后统一推送，或者单独修改每个 radosgw 服务器的配置为统一配置，然后重启 RGW 服务。

  https://docs.ceph.com/en/latest/radosgw/frontends/

 

  在ceph.conf最后面添加针对当前节点的自定义配置如下

	root@ceph-mgr1:~# vim /etc/ceph/ceph.conf
	[client.rgw.ceph-mgr1]
	rgw_host = ceph-mgr1
	rgw_frontends = civetweb port=9900

 

  重启节点 mgr1 的 radosgw 服务

	root@ceph-mgr1:~# systemctl restart ceph-radosgw@rgw.ceph-mgr1.service
	root@ceph-mgr1:~# systemctl status ceph-radosgw@rgw.ceph-mgr1.service
	ceph-radosgw@rgw.ceph-mgr1.service - Ceph rados gateway
	Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; indirect; vendor preset: enabled)
	Active: active (running) since Wed 2022-12-14 11:44:11 CST; 6s ago
	Main PID: 4196 (radosgw)
	Tasks: 603
	CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.ceph-mgr1.service
	©¸©¤4196 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph
	Dec 14 11:44:11 ceph-mgr1 systemd[1]: Started Ceph rados gateway.
	Dec 14 11:44:11 ceph-mgr1 radosgw[4196]: 2022-12-14T11:44:11.494+0800 7f76c28843c0 -1 IMPORTANT: the civetweb frontend is
	root@ceph-mgr1:~# ss -lntup|grep 9900
	tcp LISTEN 0 128 0.0.0.0:9900 0.0.0.0:* users:(("radosgw",pid=4196,fd=75))

 

 

  实现高可用

  安装haproxy并配置反向代理：

  配置haproxy，反向代理 ceph-mgr1 和 mgr2 的radosgw服务tcp网络端口，mgr2的端口此时还为默认的7480端口.

	root@haproxyA:~# vim /etc/haproxy/haproxy.cfg
	listen ceph-radosgw-8090
	bind :8090
	mode tcp
	server ceph-mgr1 192.168.100.38:9900 check inter 3s fall 3 rise 2
	server ceph-mgr2 192.168.100.39:7480 check inter 3s fall 3 rise 2
	root@haproxyA:~# systemctl restart haproxy
	root@haproxyA:~# systemctl status haproxy
	● haproxy.service - HAProxy Load Balancer
	Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
	Active: active (running) since Wed 2022-12-14 12:00:17 CST; 4s ago
	Docs: man:haproxy(1)
	file:/usr/share/doc/haproxy/configuration.txt.gz
	Process: 1401 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
	Main PID: 1413 (haproxy)
	Tasks: 2 (limit: 2236)
	Memory: 2.2M
	CGroup: /system.slice/haproxy.service
	├─1413 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
	└─1417 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
	Dec 14 12:00:17 haproxyA systemd[1]: Starting HAProxy Load Balancer...
	Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy ceph-radosgw-8090 started.
	Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy ceph-radosgw-8090 started.
	Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy statistics started.
	Dec 14 12:00:17 haproxyA haproxy[1413]: Proxy statistics started.
	Dec 14 12:00:17 haproxyA haproxy[1413]: [NOTICE] 347/120017 (1413) : New worker #1 (1417) forked
	Dec 14 12:00:17 haproxyA systemd[1]: Started HAProxy Load Balancer.
	root@haproxyA:~# ss -lntup|grep 8090
	tcp LISTEN 0 3000 0.0.0.0:8090 0.0.0.0:* users:(("haproxy",pid=1417,fd=7)

 

 

  浏览器访问haproxy代理地址 192.168.100.20:8090

 

  查看haproxy的服务代理日志，能看到将客户端的请求均衡代理到后端实际的ceph-mgr节点 endpoint

 

回到顶部

  日志及其它优化配置

  创建日志目录

	root@ceph-mgr2:~# mkdir /var/log/radosgw
	root@ceph-mgr2:~# chown ceph.ceph /var/log/radosgw
	root@ceph-mgr2:~# vim /etc/ceph/ceph.conf
	[client.rgw.ceph-mgr1]
	rgw_host = ceph-mgr1
	rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/cephrgw.pem error_log_file=/var/log/radosgw/radosgw.error.log access_log_file=/var/log/radosgw/radosgw.access.log request_timeout_ms=30000 num_threads=200"
	[client.rgw.ceph-mgr2]
	rgw_host = ceph-mgr2
	rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/cephrgw.pem error_log_file=/var/log/radosgw/radosgw.error.log access_log_file=/var/log/radosgw/radosgw.access.log request_timeout_ms=30000 num_threads=200"

 

  error_log_file： 指定radosgw错误日志路径

  access_log_file： 指定radosgw访问日志路径

  request_timeout_ms：指定radosgw访问超时时间

  num_threads： 指定radosgw运行线程数量，默认线程数是100，https://docs.ceph.com/en/mimic/radosgw/config-ref/

 

 

  重启radosgw

	root@ceph-mgr2:/etc/ceph# systemctl restart ceph-radosgw@rgw.ceph-mgr2.service
	root@ceph-mgr2:/etc/ceph# systemctl status ceph-radosgw@rgw.ceph-mgr2.service

 

  验证日志

 

回到顶部

  创建 RGW 账户

  在ceph管理节点创建对象用户

  --uid 指定用户ID

  --display-name 指定显示用户名称

	cephadmin@ceph-deploy:~/ceph-cluster$ radosgw-admin user create --uid="user1" --display-name="user1"
	{
	"user_id": "user1",
	"display_name": "user1",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"subusers": [],
	"keys": [
	{
	"user": "user1",
	"access_key": "45CMIRWTFQY9DGJX7W1Z",
	"secret_key": "EyFmlD51WWfCGbtxFYZcygwDc48QWMYyKs13nuDD"
	}
	],
	"swift_keys": [],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"default_storage_class": "",
	"placement_tags": [],
	"bucket_quota": {
	"enabled": false,
	"check_on_raw": false,
	"max_size": -1,
	"max_size_kb": 0,
	"max_objects": -1
	},
	"user_quota": {
	"enabled": false,
	"check_on_raw": false,
	"max_size": -1,
	"max_size_kb": 0,
	"max_objects": -1
	},
	"temp_url_keys": [],
	"type": "rgw",
	"mfa_ids": []
	}

  注意保存对象用户的 access_key 和 secret_key

 

  查看用户信息

	root@ceph-mgr1:/var/log/ceph# radosgw-admin user --uid="user1" info
	{
	"user_id": "user1",
	"display_name": "user1",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"subusers": [],
	"keys": [
	{
	"user": "user1",
	"access_key": "45CMIRWTFQY9DGJX7W1Z",
	"secret_key": "EyFmlD51WWfCGbtxFYZcygwDc48QWMYyKs13nuDD"
	}
	],
	"swift_keys": [],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"default_storage_class": "",
	"placement_tags": [],
	"bucket_quota": {
	"enabled": false,
	"check_on_raw": false,
	"max_size": -1,
	"max_size_kb": 0,
	"max_objects": -1
	},
	"user_quota": {
	"enabled": false,
	"check_on_raw": false,
	"max_size": -1,
	"max_size_kb": 0,
	"max_objects": -1
	},
	"temp_url_keys": [],
	"type": "rgw",
	"mfa_ids": []
	}

 

  查询所有用户

	root@ceph-mgr1:/var/log/ceph# radosgw-admin metadata list user
	[
	"user1"
	]

 

回到顶部

RGW 账户权限控制

 参考aws 官网文档介绍：https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/example-bucket-policies.html

 

  账户权限介绍

  1、授权简介和预览

  Resources： 授权的目的 Buckets、objects等资源，必须指定。

  Actions：要授予的动作，CreateBucket、DeleteObject、GetObject、PubObject。必须指定

  Effect：要授予的操作效果是允许（allow）还是拒绝（deny），默认为拒绝访问所有的资源，必须指定。

  Principal: 要授权的目的账号，必须指定

  Condition：授权策略生效的条件，比如访问TLS版本等，非必须，可不写。

	{
	“Condition”: {
	“NumericLessThan”: {
	“s3:TlsVersion”: 1.2
	}
	}
	}

 

  2、权限集合

   https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_Operations.html

 

  权限配置

  1、授予匿名用户对 bucket01 的 GetObject权限，仅可以查看桶内的文件。

  创建权限json文件

	[root@ansible ~]# vim bucket01-policy.json
	{
	"Version": "2012-10-17",
	"Statement": [
	{
	"Effect": "Allow",
	"Principal": "*",
	"Action": "s3:GetObject",
	"Resource": [
	"arn:aws:s3:::bucket01/*"
	]
	}
	]
	}

 

  进行授权

	[root@ansible ~]# s3cmd setpolicy bucket01-policy.json s3://bucket01
	s3://bucket01/: Policy updated

 

  验证权限，客户端浏览器访问 http://rgw.cncf.net/bucket01/<文件名>