VictoriaMetrics：使用vmctl来实现vm-storage向victoria-metrics-prod(单机版)迁移数据

前一篇提到了，vm-storage的备份数据，无法被victoria-metrics-prod(单机版)读取。
继续翻文档发现vmctl可以实现这个效果：

1.启动vm-restore恢复数据

vmrestore-prod \
  -configFilePath="/etc/cos/config.ini" \
  -credsFilePath="/etc/cos/creds.ini" \
  -customS3Endpoint="https://cos.ap-guangzhou.myqcloud.com" \
  -src="s3://bucket/vm-storage/2021-12-27/" \
  -storageDataPath="/var/victoria-metrics/data/2021-12-27/" \
  -httpListenAddr="0.0.0.0:8420" \
  -memory.allowedBytes=2GB \
  -concurrency=10 \
  -maxBytesPerSecond=100MB 

2.启动vm-storage

vmstorage-prod \
  -memory.allowedBytes 30GB \
  -storageDataPath /var/victoria-metrics/data/2021-12-27/ \
  -retentionPeriod=15d \
  -httpListenAddr 0.0.0.0:8482 \
  -vminsertAddr 0.0.0.0:8400 \
  -vmselectAddr 0.0.0.0:8401 \
  -search.maxTagKeys 100000 \
  -search.maxTagValues 1000000 \
  -search.maxUniqueTimeseries=10000000

3.启动一个vm-select来查询这个节点

vmselect-prod \
  -cacheDataPath /var/victoria-metrics/cache \
  -memory.allowedBytes 10GB \
  -replicationFactor 1 \
  -storageNode 127.0.0.1:8401 \
  -httpListenAddr 0.0.0.0:8481 \
  -search.maxConcurrentRequests 2000 \
  -search.maxPointsPerTimeseries 5000000 \
  -search.maxQueryDuration 600s

4.启动victoria-metrics-prod(单机版)

victoria-metrics-prod \
  -dedup.minScrapeInterval=30s \
  -httpListenAddr="0.0.0.0:8428" \
  -insert.maxQueueDuration=2m \
  -maxInsertRequestSize=500MB \
  -memory.allowedBytes=7GB \
  -retentionPeriod=1y \
  -search.maxConcurrentRequests=500 \
  -search.maxPointsPerTimeseries=100000 \
  -search.maxUniqueTimeseries=1000000 \
  -storageDataPath=/var/victoria-metrics/data/vm-single/data/

5.启动vmctl来传输数据

vmctl-prod vm-native \
  --vm-native-filter-time-start=0 \
  --vm-native-filter-time-end=$(date +"%s") \
  --vm-native-src-addr="http://127.0.0.1:8481/select/0/prometheus/" \
  --vm-native-dst-addr="http://127.0.0.1:8428/"

运行结果：

VictoriaMetrics Native import mode
Initing export pipe from "http://127.0.0.1:8481/select/0/prometheus" with filters:
        filter: match[]={__name__!=""}
        start: 0
        end: 1640919407
Initing import process to "http://127.0.0.1:8428":
Total: 5.81 GiB ↖ Speed: 1.29 MiB p/s                                                                                                                                                                                                                                       
2021/12/31 12:13:22 Total time: 1h16m35.022151869s

结论：

• 用vmctl来传输数据，奇慢，每秒传输几百KB数据。生产环境中用于迁移数据，简直是找死！
  • 统计下来，每核传输数速率 663kb/s (主要是vm单机版耗CPU很高)
  • 传输的时候，所有核跑满。可以通过增加CPU资源加快导入速度。
• 这个东西可能只适合特定的某些数据的导入导出