【磐维数据库】限制备份速度

中国移动磐维数据是基于openGauss定制开发的中国移动自用版OLTP数据库。自去2023年年12月发布以来，受到广泛关注，目前已成功上线百余套。 在产品落地的过程中，我们积累了大量的迁移、适配，以及问题分析诊断的经验。 北京海量数据技术股份有限公司，作为移动磐维集中式数据库外协厂商，对集中式磐维数据库的运维、管理、开发等均有深入了解。在江西移动现场运维整理汇总经验。

场景：
在一些场景下，为了减少备份对IO的占用，可以通过操作系统成cgroup 资源限制对备份的速度进行限制，这里以对io资源进行限制示例

1、对于一个新的备份目录初始化设置

mkdir /backup/
gs_probackup init -B /backup
gs_probackup add-instance -B /backup -D /database/panweidb/data --instance panweidb

2、在没有做资源限制下备份，可以观察到12s完成了备份

这里数据传输耗时7s

[omm@vdb1 backup]$ gs_probackup backup -B "/backup" --instance "panweidb" -b FULL -D "/database/panweidb/data" -U "omm" --threads=1
INFO: Backup start, pw_probackup version: 2.4.2, instance: panweidb, backup ID: STGPA5, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
LOG: Backup destination is initialized
LOG: This PanWeiDB instance was initialized with data block checksums. Data block corruption will be detected
LOG: Database backup start
LOG: started streaming WAL at 1/4C000000 (timeline 1)
[2025-03-21 15:10:53]: check identify system success
[2025-03-21 15:10:53]: send START_REPLICATION 1/4C000000 success
[2025-03-21 15:10:53]: keepalive message is received
[2025-03-21 15:10:53]: keepalive message is received
INFO: PGDATA size: 3240MB
INFO: Start transferring data files
LOG: Creating page header map "/backup/backups/panweidb/STGPA5/page_header_map"
[2025-03-21 15:10:58]: keepalive message is received
INFO: Data files are transferred, time elapsed: 7s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
LOG: stop_lsn: 1/4C000270
LOG: Looking for LSN 1/4C000270 in segment: 00000001000000010000004C
LOG: Found WAL segment: /backup/backups/panweidb/STGPA5/database/pg_xlog/00000001000000010000004C
LOG: Thread [0]: Opening WAL segment "/backup/backups/panweidb/STGPA5/database/pg_xlog/00000001000000010000004C"
LOG: Found LSN: 1/4C000270
[2025-03-21 15:11:05]:(null): not renaming "/backup/backups/panweidb/STGPA5/database/pg_xlog/00000001000000010000004D", segment is not complete.
LOG: finished streaming WAL at 1/4D0000B0 (timeline 1)
LOG: Getting the Recovery Time from WAL
LOG: Thread [0]: Opening WAL segment "/backup/backups/panweidb/STGPA5/database/pg_xlog/00000001000000010000004C"
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup STGPA5
INFO: Backup STGPA5 data files are valid
INFO: Backup STGPA5 resident size: 3275MB
INFO: Backup STGPA5 completed

[omm@vdb1 backup]$ gs_probackup show -B /backup

BACKUP INSTANCE 'panweidb'
==========================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Type Status
==========================================================================================================================================
panweidb 9.2 STGPA5 2025-03-21 15:11:00+08 FULL STREAM 1/0 12s 3259MB 16MB 0.99 1/4C000028 1/4C000270 FILE OK

3、配置资源限制

3.1 安装包

yum install -y libcgroup-tools

3.2 验证cgroup已经挂载

[root@vdb1 ~]# mount |grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)

3.3 创建cgroup目录

[root@vdb1 ~]# mkdir -p /sys/fs/cgroup/blkio/gsprobackup_limit

3.4 修改cgroup目录的所有者，这里数据库安装用户为omm

[root@vdb1 ~]# chown -R omm:dbgrp /sys/fs/cgroup/blkio/gsprobackup_limit/

3.5 查看备份目录所在盘的maj和minor,这里/backup 的盘的maj:minor 为253:2

[root@vdb1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 9.4G 0 rom /mnt
nvme0n1 259:0 0 80G 0 disk
├─nvme0n1p1 259:1 0 1G 0 part /boot
└─nvme0n1p2 259:2 0 79G 0 part
├─bel-root 253:0 0 50G 0 lvm /
├─bel-swap 253:1 0 4G 0 lvm
└─bel-home 253:3 0 25.1G 0 lvm /home
nvme0n2 259:3 0 20G 0 disk
└─test-testlv 253:2 0 10G 0 lvm /backup

3.6 写入限速，22428800为bytes ，表示限制速度为22428800bytes/s

echo "253:2 22428800" | sudo tee /sys/fs/cgroup/blkio/gsprobackup_limit/blkio.throttle.read_bps_device
echo "253:2 22428800" | sudo tee /sys/fs/cgroup/blkio/gsprobackup_limit/blkio.throttle.write_bps_device

4、数据库用户执行备份测试，限制后，总耗时为1m:20s

[omm@vdb1 ~]$ cgexec -g blkio:gsprobackup_limit env LD_LIBRARY_PATH=$LD_LIBRARY_PATH nocache gs_probackup backup -B "/backup" --instance "panweidb" -b FULL -D "/database/panweidb/data" -U "omm" --threads=1
env: ‘nocache’: Permission denied
[omm@vdb1 ~]$ cgexec -g blkio:gsprobackup_limit env LD_LIBRARY_PATH=$LD_LIBRARY_PATH gs_probackup backup -B "/backup" --instance "panweidb" -b FULL -D "/database/panweidb/data" -U "omm" --threads=1
INFO: Backup start, pw_probackup version: 2.4.2, instance: panweidb, backup ID: STGQ9D, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
LOG: Backup destination is initialized
LOG: This PanWeiDB instance was initialized with data block checksums. Data block corruption will be detected
LOG: Database backup start
LOG: started streaming WAL at 1/50000000 (timeline 1)
[2025-03-21 15:32:01]: check identify system success
[2025-03-21 15:32:01]: send START_REPLICATION 1/50000000 success
[2025-03-21 15:32:01]: keepalive message is received
[2025-03-21 15:32:01]: keepalive message is received
INFO: PGDATA size: 3240MB
INFO: Start transferring data files
LOG: Creating page header map "/backup/backups/panweidb/STGQ9D/page_header_map"
[2025-03-21 15:32:07]: keepalive message is received
[2025-03-21 15:32:12]: keepalive message is received
[2025-03-21 15:32:17]: keepalive message is received
[2025-03-21 15:32:22]: keepalive message is received
[2025-03-21 15:32:27]: keepalive message is received
[2025-03-21 15:32:32]: keepalive message is received
[2025-03-21 15:32:37]: keepalive message is received
[2025-03-21 15:32:42]: keepalive message is received
[2025-03-21 15:32:47]: keepalive message is received
[2025-03-21 15:32:52]: keepalive message is received
[2025-03-21 15:32:57]: keepalive message is received
[2025-03-21 15:33:02]: keepalive message is received
[2025-03-21 15:33:07]: keepalive message is received
INFO: Data files are transferred, time elapsed: 1m:8s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
LOG: stop_lsn: 1/50000390
LOG: Looking for LSN 1/50000390 in segment: 000000010000000100000050
LOG: Found WAL segment: /backup/backups/panweidb/STGQ9D/database/pg_xlog/000000010000000100000050
LOG: Thread [0]: Opening WAL segment "/backup/backups/panweidb/STGQ9D/database/pg_xlog/000000010000000100000050"
LOG: Found LSN: 1/50000390
LOG: finished streaming WAL at 1/51000000 (timeline 1)
LOG: Getting the Recovery Time from WAL
LOG: Thread [0]: Opening WAL segment "/backup/backups/panweidb/STGQ9D/database/pg_xlog/000000010000000100000050"
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 6s
INFO: Validating backup STGQ9D
INFO: Backup STGQ9D data files are valid
INFO: Backup STGQ9D resident size: 3259MB
INFO: Backup STGQ9D completed

[omm@vdb1 ~]$ gs_probackup show -B /backup

BACKUP INSTANCE 'panweidb'
============================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Type Status
============================================================================================================================================
panweidb 9.2 STGQ9D 2025-03-21 15:33:10+08 FULL STREAM 1/0 1m:20s 3243MB 16MB 1.00 1/50000028 1/50000390 FILE OK
panweidb 9.2 STGPFU 2025-03-21 15:15:33+08 FULL STREAM 1/0 1m:26s 3243MB 16MB 1.00 1/4E000028 1/4E000390 FILE OK
panweidb 9.2 STGPA5 2025-03-21 15:11:00+08 FULL STREAM 1/0 12s 3259MB 16MB 0.99 1/4C000028 1/4C000270 FILE OK

这里数据传输耗时1m:8s

总结：

通过cgroup是可以限制备份速度的。

注意：

限制io ，在用iostat 命令查看时，%iowait会变大。

使用iostat 查看io写入速度时，在写入的带宽可能会远远大于限制的速率，这是因为probackup写入会先写入内存page缓存，再从缓存写入磁盘，这种io限制只能对直接io写入磁盘有效。

iostat查看读取速度测试基本不会超过限制的速率。

在probackup校验备份阶段，需要读取备份，这时可以明显观察到速率被限制在22428800下。