Ceph集群性能基准(压力)测试
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
目录
一.rados bench工具实现基准评测Ceph性能
1.rados bench概述
rados bench是Ceph自带的性能测试工具，主要用于评估RADOS存储池的写、顺序读、随机读性能。
参考链接:
https://www.ibm.com/docs/zh/storage-ceph/7.0.0?topic=benchmark-benchmarking-ceph-performance
2.环境准备
[root@ceph141 ~]# ceph osd pool create JasonYin2020
pool 'JasonYin2020' created
[root@ceph141 ~]#
3.写性能测试
[root@ceph141 ~]# rados bench -p JasonYin2020 30 write -b 4M -t 16 --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_ceph141_76435
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 35 19 75.9116 76 0.0875339 0.45294
2 16 77 61 121.906 168 0.327283 0.455255
3 16 117 101 134.586 160 0.594012 0.436621
4 16 158 142 141.922 164 0.332936 0.411539
5 16 207 191 152.729 196 0.730204 0.40622
6 16 255 239 159.268 192 0.234069 0.387
7 16 302 286 163.366 188 0.70147 0.379653
8 16 346 330 164.942 176 0.625414 0.37525
9 16 388 372 165.278 168 0.187506 0.377352
10 16 435 419 167.543 188 0.265819 0.374599
11 16 477 461 167.58 168 0.193465 0.374191
12 16 525 509 169.611 192 0.252746 0.369712
13 16 567 551 169.454 168 0.707838 0.37072
14 16 605 589 168.178 152 0.180226 0.373712
15 16 640 624 166.293 140 0.278596 0.375678
16 16 680 664 165.895 160 0.433091 0.377307
17 16 728 712 167.427 192 0.267873 0.378908
18 16 769 753 167.143 164 0.26837 0.378465
19 16 809 793 166.764 160 0.25623 0.378377
2025-09-17T14:42:35.950863+0800 min lat: 0.0330067 max lat: 1.37957 avg lat: 0.378676
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 848 832 166.223 156 0.35235 0.378676
21 16 891 875 166.495 172 0.152146 0.379543
22 16 930 914 166.017 156 0.144854 0.379113
23 16 970 954 165.754 160 0.250361 0.380221
24 16 1012 996 165.845 168 0.736403 0.38271
25 16 1052 1036 165.61 160 0.29337 0.382724
26 16 1091 1075 165.238 156 0.688238 0.383449
27 16 1138 1122 166.076 188 0.565968 0.382761
28 16 1188 1172 167.283 200 0.288764 0.380345
29 16 1229 1213 167.166 164 0.303851 0.380236
30 16 1271 1255 167.191 168 0.464374 0.3793
Total time run: 30.1847 # 写入的时间
Total writes made: 1271 # 写入次数
Write size: 4194304 # 写入数据量大小[单位为Bytes，对应为4MB，可以使用-b选项修改哟~]
Object size: 4194304 # 对象的大小，单位为Bytes，和上面的-b选项有关，4MB
Bandwidth (MB/sec): 168.43 # 带宽（MB/秒） *****
Stddev Bandwidth: 22.8237 # 标准差带宽
Max bandwidth (MB/sec): 200 # 最大带宽
Min bandwidth (MB/sec): 76 # 最小带宽
Average IOPS: 42 # 平均IOPS（每秒读写次数）*****
Stddev IOPS: 5.70592 # 标准差IOPS
Max IOPS: 50 # 最大IOPS（每秒读写次数）
Min IOPS: 19 # 最小IOPS（每秒读写次数）：19
Average Latency(s): 0.379276 # 平均延迟（秒） *****
Stddev Latency(s): 0.223052 # 标准差延迟（秒）
Max latency(s): 1.37957 # 最大延迟（秒）
Min latency(s): 0.0330067 # 最小延迟（秒）
[root@ceph141 ~]#
[root@ceph141 ~]# rados df | egrep "JasonYin2020|POOL_NAME"
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
JasonYin2020 15 GiB 1272 0 3816 0 0 0 0 0 B 1272 5.0 GiB 0 B 0 B
[root@ceph141 ~]#
[root@ceph141 ~]# echo 4194304/1024/1024|bc
4
[root@ceph141 ~]#
[root@ceph141 ~]# rados bench -p JasonYin2020 30 write -b 8M -t 16 --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 8388608 bytes to objects of size 8388608 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_ceph141_90192
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 32 17 135.899 136 0.484957 0.579675
2 15 55 40 159.907 184 0.276308 0.599671
3 15 79 64 170.485 192 1.06627 0.673351
4 15 102 87 173.853 184 0.261028 0.629182
5 15 129 114 182.26 216 0.321719 0.646102
6 15 155 140 186.542 208 1.42941 0.636997
7 15 183 168 191.841 224 1.30052 0.617982
8 15 212 197 196.849 232 0.538708 0.617369
9 15 241 226 200.744 232 0.464318 0.608606
10 15 269 254 203.058 224 0.700391 0.609036
11 15 297 282 204.956 224 0.754455 0.604194
12 15 326 311 207.19 232 0.443682 0.597935
13 15 352 337 207.239 208 0.951104 0.597437
14 15 382 367 209.575 240 0.520775 0.594501
15 15 408 393 209.459 208 0.265721 0.593673
16 15 434 419 209.365 208 0.382622 0.594403
17 15 459 444 208.787 200 0.556966 0.597566
18 15 478 463 205.631 152 0.396002 0.605588
19 15 501 486 204.453 184 0.900571 0.609092
2025-09-17T16:05:46.783097+0800 min lat: 0.14777 max lat: 1.86189 avg lat: 0.605509
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 15 532 517 206.626 248 0.758761 0.605509
21 15 560 545 207.45 224 0.43028 0.607474
22 15 589 574 208.564 232 0.515777 0.603638
23 15 614 599 208.165 200 0.723795 0.60354
24 15 643 628 209.154 232 0.625574 0.602199
25 15 668 653 208.786 200 0.623777 0.600819
26 15 695 680 209.058 216 0.288917 0.600689
27 15 725 710 210.2 240 0.520903 0.601474
28 15 748 733 209.264 184 0.736457 0.600513
29 15 778 763 210.322 240 0.42654 0.599972
30 12 805 793 211.306 240 0.649407 0.600597
Total time run: 30.3276
Total writes made: 805
Write size: 8388608
Object size: 8388608
Bandwidth (MB/sec): 212.347
Stddev Bandwidth: 26.5482
Max bandwidth (MB/sec): 248
Min bandwidth (MB/sec): 136
Average IOPS: 26
Stddev IOPS: 3.31853
Max IOPS: 31
Min IOPS: 17
Average Latency(s): 0.598054
Stddev Latency(s): 0.252862
Max latency(s): 1.86189
Min latency(s): 0.14777
[root@ceph141 ~]#
4.顺序读测试
[root@ceph141 ~]# rados bench -p JasonYin2020 10 seq -t 16
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 148 132 527.852 528 0.0209347 0.0963214
2 16 286 270 539.841 552 0.466894 0.108418
3 16 408 392 521.602 488 0.0701871 0.114623
4 16 540 524 523.18 528 0.0244368 0.11458
5 16 665 649 518.47 500 0.111208 0.118413
6 16 796 780 519.38 524 0.0285209 0.11887
7 16 911 895 510.895 460 0.368056 0.121244
8 16 1033 1017 508.012 488 0.47419 0.12218
9 16 1144 1128 500.649 444 0.178578 0.124281
10 11 1268 1257 502.165 516 0.0188324 0.124061
Total time run: 10.1118
Total reads made: 1268
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 501.591 # 不难发现，咱们的ceph集群顺序读取性能要比写入性能高。
Average IOPS: 125
Stddev IOPS: 8.35397
Max IOPS: 138
Min IOPS: 111
Average Latency(s): 0.125749
Max latency(s): 0.659622
Min latency(s): 0.0059344
[root@ceph141 ~]#
[root@ceph141 ~]# echo 8388608/1024/1024|bc
8
[root@ceph141 ~]#
5.随机读测试
[root@ceph141 ~]# echo 3 | sudo tee /proc/sys/vm/drop_caches && sudo sync # 随机读取之前要清空缓存。
3
[root@ceph141 ~]#
[root@ceph141 ~]# rados bench -p JasonYin2020 10 rand -t 16
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 135 119 475.875 476 0.154917 0.109734
2 16 283 267 533.335 592 0.0805749 0.110209
3 16 423 407 542.194 560 0.0498927 0.112117
4 16 557 541 540.616 536 0.0656441 0.112463
5 16 685 669 534.851 512 0.0896034 0.113753
6 16 799 783 521.596 456 0.0356925 0.118029
7 16 925 909 519.074 504 0.014017 0.119841
8 16 1037 1021 510.086 448 0.0280757 0.121386
9 16 1153 1137 504.938 464 0.13386 0.123689
10 13 1290 1277 510.415 560 0.00783718 0.122106
Total time run: 10.1043
Total reads made: 1290
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 510.675 # 不难发现，随机读性能也不差！
Average IOPS: 127
Stddev IOPS: 12.4904
Max IOPS: 148
Min IOPS: 112
Average Latency(s): 0.123349
Max latency(s): 0.648618
Min latency(s): 0.00488359
[root@ceph141 ~]#
6.清理测试数据
[root@ceph141 ~]# rados df | egrep "JasonYin2020|POOL_NAME"
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
JasonYin2020 15 GiB 1272 0 3816 0 0 0 5398 21 GiB 1272 5.0 GiB 0 B 0 B
[root@ceph141 ~]#
[root@ceph141 ~]# rados -p JasonYin2020 cleanup
Removed 1271 objects
[root@ceph141 ~]#
[root@ceph141 ~]# rados df | egrep "JasonYin2020|POOL_NAME"
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
JasonYin2020 0 B 0 0 0 0 0 0 5399 21 GiB 2544 5.0 GiB 0 B 0 B
[root@ceph141 ~]#
二.rbd bench工具基准测试Ceph块性能
1.基准测试 Ceph 块性能测试
Ceph 包含 rbd bench-write 命令，用于测试对块设备的顺序写入，以测量吞吐量和等待时间。
缺省字节大小为 4096 ，缺省 I/O 线程数为 16 ，要写入的缺省总字节数为 1 GB。 这些缺省值可分别由 --io-size， --io-threads 和 --io-total 选项修改。
参考链接:
https://www.ibm.com/docs/zh/storage-ceph/7.0.0?topic=benchmark-benchmarking-ceph-block-performance
2.创建测试块设备
[root@ceph141 ~]# rbd ls -l -p JasonYin2020
[root@ceph141 ~]#
[root@ceph141 ~]# rbd create --size 10G JasonYin2020/linux99
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -l -p JasonYin2020
NAME SIZE PARENT FMT PROT LOCK
linux99 10 GiB 2
[root@ceph141 ~]#
3.块设备写性能测试案例
[root@ceph141 ~]# rbd bench --io-type write linux99 --pool=JasonYin2020
bench type write io_size 4096 io_threads 16 bytes 1073741824 pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 27168 27293.2 107 MiB/s
2 58000 29066.1 114 MiB/s
3 88448 29527.4 115 MiB/s
4 118624 29689.7 116 MiB/s
5 148672 29737.6 116 MiB/s
6 175168 29600 116 MiB/s
7 206304 29660.8 116 MiB/s
8 238144 29939.2 117 MiB/s
elapsed: 8 ops: 262144 ops/sec: 29993.6 bytes/sec: 117 MiB/s
[root@ceph141 ~]#
4.块设备读性能测试
[root@ceph141 ~]# rbd bench --io-type read linux99 --pool=JasonYin2020
bench type read io_size 4096 io_threads 16 bytes 1073741824 pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 8672 8722.88 34 MiB/s
2 16896 8472.94 33 MiB/s
3 24944 8331.1 33 MiB/s
4 32992 8260.25 32 MiB/s
5 39904 7990.38 31 MiB/s
6 46144 7494.39 29 MiB/s
7 52896 7199.99 28 MiB/s
8 61200 7251.19 28 MiB/s
9 69520 7305.59 29 MiB/s
10 77664 7551.99 29 MiB/s
11 85904 7951.99 31 MiB/s
12 94144 8249.59 32 MiB/s
13 101888 8137.59 32 MiB/s
14 109600 8015.99 31 MiB/s
15 116784 7823.99 31 MiB/s
16 124096 7638.39 30 MiB/s
17 131632 7497.59 29 MiB/s
18 138528 7327.99 29 MiB/s
19 144784 7036.79 27 MiB/s
20 151936 7030.39 27 MiB/s
21 159344 7049.59 28 MiB/s
22 167744 7222.39 28 MiB/s
23 177200 7734.39 30 MiB/s
24 184656 7974.39 31 MiB/s
25 191344 7881.59 31 MiB/s
26 198896 7910.39 31 MiB/s
27 206736 7798.39 30 MiB/s
28 215616 7683.19 30 MiB/s
29 224176 7903.99 31 MiB/s
30 233600 8451.19 33 MiB/s
31 242512 8723.19 34 MiB/s
32 251344 8921.59 35 MiB/s
33 258992 8675.19 34 MiB/s
elapsed: 33 ops: 262144 ops/sec: 7847.67 bytes/sec: 31 MiB/s
[root@ceph141 ~]#
三.FIO工具测试CephFS基准测试【建议使用scp测试，发现fio测试需要进一步研究】
1.FIO概述
使用 FIO 工具对 Ceph File System (CephFS) 性能进行基准测试。
参考链接:
https://www.ibm.com/docs/zh/storage-ceph/7.0.0?topic=benchmark-benchmarking-cephfs-performance
2.安装fio程序
[root@ceph143 ~]# apt -y install fio
3.挂载cephfs
1.服务端创建用户
[root@ceph141 ~]# ceph auth add client.cephfs mon 'allow r' mds 'allow rw' osd 'allow rwx'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph auth get client.cephfs
[client.cephfs]
key = AQAu3shoEApwHBAAtzhU7vNjtormRMyAR7wINw==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rwx"
[root@ceph141 ~]#
2.客户端挂载
[root@ceph143 ~]# mkdir /data
[root@ceph143 ~]# mount -t ceph 10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ /data -o name=cephfs,secret=AQAu3shoEApwHBAAtzhU7vNjtormRMyAR7wINw==
[root@ceph143 ~]#
[root@ceph143 ~]# df -h | grep data
10.0.0.141:6789,10.0.0.142:6789,10.0.0.143:6789:/ 1.7T 280M 1.7T 1% /data
[root@ceph143 ~]#
4.运行 FIO 命令测试验证
从4k开始bs值，并以2个增量重复 (4k， 8k， 16k， 32k ... 128k... 512k， 1 m ， 2 m ， 4 m) 以及不同的iodepth设置。
还可按期望的工作负载操作大小运行测试。例如，具有不同 iodepth 值的 4 K 检验:
测试代码:
fio --name=randwrite --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=1 --size=5G --runtime=60 --group_reporting=1
fio --name=randwrite1 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=32 --size=5G --runtime=60 --group_reporting=1
fio --name=randwrite2 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=64 --size=5G --runtime=60 --group_reporting=1
fio --name=randwrite3 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=128 --size=5G --runtime=60 --group_reporting=1
实战演练：
[root@ceph143 ~]# cd /data/
[root@ceph143 data]#
[root@ceph143 data]# ll
total 4
drwxr-xr-x 5 root root 3 Sep 16 15:21 ./
drwxr-xr-x 22 root root 4096 Sep 17 11:51 ../
drwxr-xr-x 2 root root 2 Sep 16 14:56 haha/
drwxr-xr-x 3 root root 1 Sep 16 15:21 linux99/
drwxr-xr-x 2 root root 2 Sep 16 14:57 xixi/
[root@ceph143 data]#
[root@ceph143 data]# fio --name=randwrite --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=1 --size=5G --runtime=60 --group_reporting=1
_reporting=1randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1160KiB/s][w=290 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=38287: Wed Sep 17 11:54:55 2025
write: IOPS=294, BW=1177KiB/s (1205kB/s)(69.0MiB/60002msec); 0 zone resets
slat (usec): min=11, max=7400, avg=24.96, stdev=61.94
clat (usec): min=549, max=405146, avg=3368.37, stdev=3833.47
lat (usec): min=1007, max=405166, avg=3393.69, stdev=3834.47
clat percentiles (usec):
| 1.00th=[ 1156], 5.00th=[ 1270], 10.00th=[ 1336], 20.00th=[ 1467],
| 30.00th=[ 1614], 40.00th=[ 1876], 50.00th=[ 2311], 60.00th=[ 3032],
| 70.00th=[ 3949], 80.00th=[ 5342], 90.00th=[ 7177], 95.00th=[ 8225],
| 99.00th=[10290], 99.50th=[11076], 99.90th=[13304], 99.95th=[13829],
| 99.99th=[17171]
bw ( KiB/s): min= 335, max= 1368, per=100.00%, avg=1178.35, stdev=103.63, samples=119
iops : min= 83, max= 342, avg=294.57, stdev=25.96, samples=119
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=43.43%, 4=27.23%, 10=28.09%, 20=1.22%, 500=0.01%
cpu : usr=0.56%, sys=0.61%, ctx=17718, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,17653,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1177KiB/s (1205kB/s), 1177KiB/s-1177KiB/s (1205kB/s-1205kB/s), io=69.0MiB (72.3MB), run=60002-60002msec
[root@ceph143 data]#
[root@ceph143 data]# fio --name=randwrite1 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=32 --size=5G --runtime=60 --group_reporting=1
randwrite1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
randwrite1: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=8860KiB/s][w=2215 IOPS][eta 00m:00s]
randwrite1: (groupid=0, jobs=1): err= 0: pid=38302: Wed Sep 17 11:55:55 2025
write: IOPS=2047, BW=8189KiB/s (8386kB/s)(480MiB/60008msec); 0 zone resets
slat (usec): min=2, max=15080, avg=18.51, stdev=57.92
clat (usec): min=1142, max=114219, avg=15607.09, stdev=9487.44
lat (usec): min=1166, max=114249, avg=15625.90, stdev=9488.41
clat percentiles (usec):
| 1.00th=[ 2245], 5.00th=[ 3359], 10.00th=[ 4555], 20.00th=[ 7046],
| 30.00th=[ 9372], 40.00th=[11731], 50.00th=[14222], 60.00th=[16712],
| 70.00th=[19530], 80.00th=[23200], 90.00th=[28181], 95.00th=[32637],
| 99.00th=[43254], 99.50th=[47973], 99.90th=[62653], 99.95th=[69731],
| 99.99th=[83362]
bw ( KiB/s): min= 5720, max= 9184, per=99.99%, avg=8188.44, stdev=686.83, samples=119
iops : min= 1430, max= 2296, avg=2047.13, stdev=171.73, samples=119
lat (msec) : 2=0.43%, 4=7.19%, 10=24.92%, 20=38.87%, 50=28.19%
lat (msec) : 100=0.40%, 250=0.01%
cpu : usr=1.07%, sys=4.55%, ctx=75674, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,122852,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=8189KiB/s (8386kB/s), 8189KiB/s-8189KiB/s (8386kB/s-8386kB/s), io=480MiB (503MB), run=60008-60008msec
[root@ceph143 data]#
[root@ceph143 data]# fio --name=randwrite2 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=64 --size=5G --runtime=60 --group_reporting=1
randwrite2: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
randwrite2: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=9876KiB/s][w=2469 IOPS][eta 00m:00s]
randwrite2: (groupid=0, jobs=1): err= 0: pid=38320: Wed Sep 17 11:56:56 2025
write: IOPS=2233, BW=8935KiB/s (9149kB/s)(524MiB/60007msec); 0 zone resets
slat (usec): min=2, max=3319, avg=18.63, stdev=33.53
clat (usec): min=1200, max=151011, avg=28626.99, stdev=18202.94
lat (usec): min=1223, max=151030, avg=28645.92, stdev=18204.03
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 8], 20.00th=[ 12],
| 30.00th=[ 17], 40.00th=[ 22], 50.00th=[ 27], 60.00th=[ 32],
| 70.00th=[ 37], 80.00th=[ 43], 90.00th=[ 53], 95.00th=[ 62],
| 99.00th=[ 83], 99.50th=[ 92], 99.90th=[ 114], 99.95th=[ 121],
| 99.99th=[ 138]
bw ( KiB/s): min= 5240, max=10824, per=99.92%, avg=8928.57, stdev=970.93, samples=119
iops : min= 1310, max= 2706, avg=2232.14, stdev=242.73, samples=119
lat (msec) : 2=0.16%, 4=3.37%, 10=12.22%, 20=21.15%, 50=50.95%
lat (msec) : 100=11.87%, 250=0.28%
cpu : usr=1.29%, sys=4.44%, ctx=71681, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,134039,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=8935KiB/s (9149kB/s), 8935KiB/s-8935KiB/s (9149kB/s-9149kB/s), io=524MiB (549MB), run=60007-60007msec
[root@ceph143 data]#
[root@ceph143 data]# fio --name=randwrite3 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=128 --size=5G --runtime=60 --group_reporting=1
randwrite3: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.28
Starting 1 process
randwrite3: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=0): [f(1)][100.0%][w=9.89MiB/s][w=2532 IOPS][eta 00m:00s]
randwrite3: (groupid=0, jobs=1): err= 0: pid=38533: Wed Sep 17 11:57:59 2025
write: IOPS=2500, BW=9.77MiB/s (10.2MB/s)(586MiB/60015msec); 0 zone resets
slat (usec): min=2, max=2945, avg=14.71, stdev=17.47
clat (usec): min=1314, max=428679, avg=51171.65, stdev=36322.58
lat (usec): min=1328, max=428683, avg=51186.60, stdev=36322.52
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 11], 20.00th=[ 20],
| 30.00th=[ 28], 40.00th=[ 37], 50.00th=[ 46], 60.00th=[ 56],
| 70.00th=[ 66], 80.00th=[ 78], 90.00th=[ 96], 95.00th=[ 114],
| 99.00th=[ 163], 99.50th=[ 197], 99.90th=[ 288], 99.95th=[ 334],
| 99.99th=[ 401]
bw ( KiB/s): min= 3184, max=11840, per=100.00%, avg=10002.52, stdev=1354.82, samples=119
iops : min= 796, max= 2960, avg=2500.61, stdev=338.72, samples=119
lat (msec) : 2=0.10%, 4=2.10%, 10=7.19%, 20=11.34%, 50=33.50%
lat (msec) : 100=37.31%, 250=8.25%, 500=0.21%
cpu : usr=1.37%, sys=4.12%, ctx=65230, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,150044,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=9.77MiB/s (10.2MB/s), 9.77MiB/s-9.77MiB/s (10.2MB/s-10.2MB/s), io=586MiB (615MB), run=60015-60015msec
[root@ceph143 data]#
[root@ceph143 data]# ll -h
total 20G
drwxr-xr-x 5 root root 7 Sep 17 11:56 ./
drwxr-xr-x 22 root root 4.0K Sep 17 11:51 ../
drwxr-xr-x 2 root root 2 Sep 16 14:56 haha/
drwxr-xr-x 3 root root 1 Sep 16 15:21 linux99/
-rw-r--r-- 1 root root 5.0G Sep 17 11:54 randwrite.0.0
-rw-r--r-- 1 root root 5.0G Sep 17 11:55 randwrite1.0.0
-rw-r--r-- 1 root root 5.0G Sep 17 11:56 randwrite2.0.0
-rw-r--r-- 1 root root 5.0G Sep 17 11:57 randwrite3.0.0
drwxr-xr-x 2 root root 2 Sep 16 14:57 xixi/
[root@ceph143 data]#
[root@ceph143 data]#
Run status group 0 (all jobs): 相关参数说明:
WRITE:
表示写的参数信息。
bw:
表示写入的速度。
io：
产生的数据大小。
run:
运行的时间。
5.其他测试
具有不同 iodepth 值的 8 K 检验:
相关代码:
fio --name=randwrite0 --rw=randwrite --direct=1 --ioengine=libaio --bs=8k --iodepth=1 --size=5G --runtime=60 --group_reporting=1
fio --name=randwrite11 --rw=randwrite --direct=1 --ioengine=libaio --bs=8k --iodepth=32 --size=5G --runtime=60 --group_reporting=1
fio --name=randwrite22 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=64 --size=5G --runtime=60 --group_reporting=1
fio --name=randwrite33 --rw=randwrite --direct=1 --ioengine=libaio --bs=4k --iodepth=128 --size=5G --runtime=60 --group_reporting=1
关键字段说明:
--name: 指定文件的名称。
--rw: 指定写入的类型，有效值为: write/randwrite/rw/randrw/trim/randtrim/trimwrite。
--direct和--ioengine选项一般可以关联设置，其中io引擎有以下有效值:
[root@ceph143 ~]# man fio
...
I/O engine
ioengine=str
Defines how the job issues I/O to the file. The following types are defined:
sync Basic read(2) or write(2) I/O. lseek(2) is used to position the I/O location. See fsync and fdatasync for syncing write I/Os.
psync Basic pread(2) or pwrite(2) I/O. Default on all supported operating systems except for Windows.
vsync Basic readv(2) or writev(2) I/O. Will emulate queuing by coalescing adjacent I/Os into a single submission.
pvsync Basic preadv(2) or pwritev(2) I/O.
pvsync2
Basic preadv2(2) or pwritev2(2) I/O.
libaio Linux native asynchronous I/O. Note that Linux may only support queued behavior with non-buffered I/O (set `direct=1' or `buffered=0'). This engine defines
engine specific options.
...
--bs:
一次读写的数据大小。
--iodepth:
指定IO的深度，计算时应该和bs有关。
--size:
测试数据量的大小。
--runtime
指定运行的时间，单位为秒。
--group_reporting
以组为生成报告。
查看文档可参考帮助信息:
[root@ceph143 ~]# man fio
彩蛋1：
[root@ceph143 data]# fio --name=yinzhengjie-write --rw=write --direct=1 --ioengine=libaio --bs=8k --iodepth=1 --size=10G --runtime=30 --group_reporting=1
yinzhengjie-write: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
yinzhengjie-write: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=3139KiB/s][w=392 IOPS][eta 00m:00s]
yinzhengjie-write: (groupid=0, jobs=1): err= 0: pid=78153: Wed Sep 17 15:44:54 2025
write: IOPS=330, BW=2645KiB/s (2708kB/s)(77.5MiB/30001msec); 0 zone resets
slat (usec): min=11, max=430, avg=26.14, stdev=12.04
clat (usec): min=945, max=18347, avg=2992.92, stdev=1857.03
lat (usec): min=957, max=18373, avg=3019.48, stdev=1859.18
clat percentiles (usec):
| 1.00th=[ 1123], 5.00th=[ 1303], 10.00th=[ 1467], 20.00th=[ 1811],
| 30.00th=[ 2024], 40.00th=[ 2212], 50.00th=[ 2409], 60.00th=[ 2638],
| 70.00th=[ 2999], 80.00th=[ 3654], 90.00th=[ 5407], 95.00th=[ 7439],
| 99.00th=[ 9896], 99.50th=[10814], 99.90th=[13304], 99.95th=[14222],
| 99.99th=[18220]
bw ( KiB/s): min= 2208, max= 4096, per=100.00%, avg=2651.37, stdev=466.69, samples=59
iops : min= 276, max= 512, avg=331.39, stdev=58.35, samples=59
lat (usec) : 1000=0.03%
lat (msec) : 2=29.11%, 4=54.26%, 10=15.70%, 20=0.91%
cpu : usr=0.24%, sys=1.20%, ctx=9929, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,9919,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=2645KiB/s (2708kB/s), 2645KiB/s-2645KiB/s (2708kB/s-2708kB/s), io=77.5MiB (81.3MB), run=30001-30001msec
[root@ceph143 data]#
[root@ceph143 data]# ll yinzhengjie-write.0.0
-rw-r--r-- 1 root root 81256448 Sep 17 15:44 yinzhengjie-write.0.0
[root@ceph143 data]#
[root@ceph143 data]# ll -h yinzhengjie-write.0.0
-rw-r--r-- 1 root root 78M Sep 17 15:44 yinzhengjie-write.0.0
[root@ceph143 data]#
彩蛋2：
[root@ceph143 data]# fio --name=yinzhengjie-write-read --rw=rw --direct=1 --ioengine=libaio --bs=8k --iodepth=1 --size=10G --runtime=30 --group_reporting=1
yinzhengjie-write-read: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
yinzhengjie-write-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=3217KiB/s,w=3241KiB/s][r=402,w=405 IOPS][eta 00m:00s]
yinzhengjie-write-read: (groupid=0, jobs=1): err= 0: pid=78408: Wed Sep 17 15:48:36 2025
read: IOPS=250, BW=2001KiB/s (2049kB/s)(58.6MiB/30003msec)
slat (usec): min=7, max=257, avg=19.29, stdev=12.35
clat (usec): min=97, max=10374, avg=958.61, stdev=1065.27
lat (usec): min=106, max=10397, avg=978.22, stdev=1066.59
clat percentiles (usec):
| 1.00th=[ 155], 5.00th=[ 210], 10.00th=[ 269], 20.00th=[ 334],
| 30.00th=[ 383], 40.00th=[ 424], 50.00th=[ 461], 60.00th=[ 519],
| 70.00th=[ 652], 80.00th=[ 1696], 90.00th=[ 2802], 95.00th=[ 3097],
| 99.00th=[ 4555], 99.50th=[ 5276], 99.90th=[ 6521], 99.95th=[ 6849],
| 99.99th=[10421]
bw ( KiB/s): min= 1408, max= 3840, per=98.97%, avg=1980.27, stdev=544.85, samples=59
iops : min= 176, max= 480, avg=247.51, stdev=68.11, samples=59
write: IOPS=251, BW=2014KiB/s (2063kB/s)(59.0MiB/30003msec); 0 zone resets
slat (usec): min=9, max=671, avg=23.66, stdev=16.24
clat (usec): min=964, max=18388, avg=2967.85, stdev=2052.60
lat (usec): min=977, max=18420, avg=2991.87, stdev=2055.08
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1237], 10.00th=[ 1336], 20.00th=[ 1549],
| 30.00th=[ 1778], 40.00th=[ 2008], 50.00th=[ 2245], 60.00th=[ 2507],
| 70.00th=[ 2966], 80.00th=[ 3818], 90.00th=[ 6128], 95.00th=[ 7832],
| 99.00th=[10159], 99.50th=[10814], 99.90th=[13566], 99.95th=[13960],
| 99.99th=[18482]
bw ( KiB/s): min= 1261, max= 3568, per=98.79%, avg=1990.32, stdev=525.10, samples=59
iops : min= 157, max= 446, avg=248.76, stdev=65.66, samples=59
lat (usec) : 100=0.01%, 250=3.80%, 500=24.78%, 750=7.86%, 1000=1.35%
lat (msec) : 2=22.84%, 4=29.17%, 10=9.64%, 20=0.55%
cpu : usr=0.31%, sys=1.44%, ctx=15394, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7503,7555,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2001KiB/s (2049kB/s), 2001KiB/s-2001KiB/s (2049kB/s-2049kB/s), io=58.6MiB (61.5MB), run=30003-30003msec
WRITE: bw=2014KiB/s (2063kB/s), 2014KiB/s-2014KiB/s (2063kB/s-2063kB/s), io=59.0MiB (61.9MB), run=30003-30003msec
[root@ceph143 data]#
[root@ceph143 data]# fio --name=yinzhengjie-write-read --rw=rw --direct=1 --ioengine=libaio --bs=8k --iodepth=1 --size=10G --runtime=60 --group_reporting=1
yinzhengjie-write-read: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=1705KiB/s,w=1913KiB/s][r=213,w=239 IOPS][eta 00m:00s]
yinzhengjie-write-read: (groupid=0, jobs=1): err= 0: pid=78433: Wed Sep 17 15:50:37 2025
read: IOPS=276, BW=2209KiB/s (2262kB/s)(129MiB/60002msec)
slat (usec): min=6, max=579, avg=16.90, stdev= 9.90
clat (usec): min=69, max=11063, avg=923.42, stdev=1122.67
lat (usec): min=76, max=11079, avg=940.62, stdev=1124.67
clat percentiles (usec):
| 1.00th=[ 126], 5.00th=[ 176], 10.00th=[ 206], 20.00th=[ 265],
| 30.00th=[ 310], 40.00th=[ 359], 50.00th=[ 400], 60.00th=[ 445],
| 70.00th=[ 553], 80.00th=[ 1778], 90.00th=[ 2769], 95.00th=[ 3032],
| 99.00th=[ 5145], 99.50th=[ 5735], 99.90th=[ 6915], 99.95th=[ 7373],
| 99.99th=[ 8455]
bw ( KiB/s): min= 1248, max= 4400, per=100.00%, avg=2212.31, stdev=748.52, samples=119
iops : min= 156, max= 550, avg=276.49, stdev=93.59, samples=119
write: IOPS=276, BW=2210KiB/s (2263kB/s)(130MiB/60002msec); 0 zone resets
slat (usec): min=8, max=1266, avg=20.61, stdev=14.21
clat (usec): min=866, max=19189, avg=2651.57, stdev=1893.19
lat (usec): min=878, max=19219, avg=2672.51, stdev=1895.37
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1156], 10.00th=[ 1254], 20.00th=[ 1418],
| 30.00th=[ 1614], 40.00th=[ 1827], 50.00th=[ 2024], 60.00th=[ 2278],
| 70.00th=[ 2606], 80.00th=[ 3228], 90.00th=[ 5145], 95.00th=[ 7177],
| 99.00th=[ 9765], 99.50th=[10814], 99.90th=[13173], 99.95th=[15926],
| 99.99th=[18744]
bw ( KiB/s): min= 1536, max= 4112, per=100.00%, avg=2214.48, stdev=719.96, samples=119
iops : min= 192, max= 514, avg=276.76, stdev=90.03, samples=119
lat (usec) : 100=0.24%, 250=8.24%, 500=25.02%, 750=3.41%, 1000=1.12%
lat (msec) : 2=27.07%, 4=26.78%, 10=7.72%, 20=0.40%
cpu : usr=0.68%, sys=1.05%, ctx=33273, majf=0, minf=17
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=16567,16576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2209KiB/s (2262kB/s), 2209KiB/s-2209KiB/s (2262kB/s-2262kB/s), io=129MiB (136MB), run=60002-60002msec
WRITE: bw=2210KiB/s (2263kB/s), 2210KiB/s-2210KiB/s (2263kB/s-2263kB/s), io=130MiB (136MB), run=60002-60002msec
[root@ceph143 data]#
后记:
如果你觉得fio工具不太好用，可以直接使用scp命令和cephfs路径的进行大文件数据拷贝，也能间接体现出cephfs的性能。
四.Ceph Object Gateway 性能进行基准测试
1.对象存储网关的性能测试
使用 s3cmd 工具对 Ceph Object Gateway 性能进行基准测试。
参考链接:
https://www.ibm.com/docs/zh/storage-ceph/7.0.0?topic=benchmark-benchmarking-ceph-object-gateway-performance
2.上载文件并测量速度。
time 命令用于度量上载的持续时间。
[root@ceph141 ~]# s3cmd ls s3://yinzhengjie
2025-09-17 02:35 39451064 s3://yinzhengjie/containerd
2025-09-17 02:36 58246432 s3://yinzhengjie/dockerd
[root@ceph141 ~]#
[root@ceph141 ~]# ll yinzhengjie-ceph-v19.tar.gz
-rw-r--r-- 1 root root 1333207552 May 16 09:38 yinzhengjie-ceph-v19.tar.gz
[root@ceph141 ~]#
[root@ceph141 ~]# ll -h yinzhengjie-ceph-v19.tar.gz
-rw-r--r-- 1 root root 1.3G May 16 09:38 yinzhengjie-ceph-v19.tar.gz
[root@ceph141 ~]#
[root@ceph141 ~]# time s3cmd put yinzhengjie-ceph-v19.tar.gz s3://yinzhengjie
upload: 'yinzhengjie-ceph-v19.tar.gz' -> 's3://yinzhengjie/yinzhengjie-ceph-v19.tar.gz' [part 1 of 85, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 24.22 MB/s done
...
upload: 'yinzhengjie-ceph-v19.tar.gz' -> 's3://yinzhengjie/yinzhengjie-ceph-v19.tar.gz' [part 85 of 85, 11MB] [1 of 1]
12001792 of 12001792 100% in 0s 53.52 MB/s done
real 0m31.253s # 命令执行的耗时时间，将近32s。
user 0m3.156s
sys 0m10.206s
[root@ceph141 ~]#
[root@ceph141 ~]# echo 1333207552/32/1024/1024|bc # 不难发现，rgw的写性能应该为39MB/s。
39
[root@ceph141 ~]#
3.下载文件并测量速度。
time 命令用于度量下载持续时间。
[root@ceph141 ~]# time s3cmd get s3://yinzhengjie/yinzhengjie-ceph-v19.tar.gz /tmp/
download: 's3://yinzhengjie/yinzhengjie-ceph-v19.tar.gz' -> '/tmp/yinzhengjie-ceph-v19.tar.gz' [1 of 1]
1333207552 of 1333207552 100% in 11s 113.77 MB/s done
real 0m11.336s # 耗时讲解12s
user 0m1.425s
sys 0m2.944s
[root@ceph141 ~]#
[root@ceph141 ~]# ll /tmp/yinzhengjie-ceph-v19.tar.gz
-rw-r--r-- 1 root root 1333207552 Sep 17 08:45 /tmp/yinzhengjie-ceph-v19.tar.gz
[root@ceph141 ~]#
[root@ceph141 ~]# echo 1333207552/12/1024/1024|bc # 最终得出结论，rgw的读性能将近105MB/s。
105
[root@ceph141 ~]#
4.列出指定存储区中的所有对象并测量响应时间。
[root@ceph141 ~]# time s3cmd ls s3://yinzhengjie/
2025-09-17 02:35 39451064 s3://yinzhengjie/containerd
2025-09-17 02:36 58246432 s3://yinzhengjie/dockerd
2025-09-17 08:45 1333207552 s3://yinzhengjie/yinzhengjie-ceph-v19.tar.gz
real 0m0.200s
user 0m0.106s
sys 0m0.026s
[root@ceph141 ~]#
本文来自博客园，作者：尹正杰，转载请注明原文链接：https://www.cnblogs.com/yinzhengjie/p/19084869，个人微信: "JasonYin2020"（添加时请备注来源及意图备注，有偿付费）
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
浙公网安备 33010602011771号