elastic-dump迁移es集群

elastic dump安装

      npm install -g cnpm --registry=https://registry.npm.taobao.org
      npm install elasticdump -g

elastic dump基本使用

           

           

           elasticdump --input=https://elastic:'111@tn111333'@10.246.181.6:9200/prob_file_description_202312 --input-cert=/home/secure/canew.crt --limit=80000  --output-cert=/home/secure/ca.crt --output=https://elastic:'444444@2024'@10.30.98.107:9200/full_flow_file_202312

           dump ended with error (get phase) => Error: unable to verify the first certificate

       

https集群迁移

      NODE_TLS_REJECT_UNAUTHORIZED=0

      NODE_TLS_REJECT_UNAUTHORIZED=0  elasticdump --input=https://elastic:'111@tn111333'@10.246.181.6:9200/prob_file_description_202312 --input-cert=/home/secure/canew.crt --limit=80000  --output-cert=/home/secure/ca.crt --output=https://elastic:'444444@2024'@10.30.98.107:9200/full_flow_file_202312

工具配置忽略证书

         

           

          unable to verify the first certificate

          自签名证书默认是不受信任的证书,一般可以通过选项参数设置是否忽略证书验证

迁移数据状态监控

curl --insecure -u elastic:111111 -XPUT 'https://10.30.92.79:9200/event_20240506/_settings' -H 'Content-Type: application/json' -d '{
        "max_result_window":35000
}'

 NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input=https://elastic:'111111'@10.30.92.71:9200/xinan_data_20240509 --input-cert=/home/secure/ca/ca.crt --limit=30000  --output-cert=/home/secure/ca/ca7.crt --output=https://elastic:'11111'@119.88.88.88:9200 > event_20240506.log  2>&1 

ctrl+z
jobs
bg %1
disown -h %1

tail -f event_20240506.log

          

         

         迁移监控

           迁移时跳过错误数据 否则遇到错误后同步程序会自动终止

            --ignore-errors

         NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input=https://elastic:11111@10.30.92.88:9200/label_20240302 --input-cert=/home/secure/ca/ca.crt --limit=10000 --ignore-errors --output-cert=/home/secure/ca/ca7.crt --output=https://elastic:222222@119.89.99.89:9200 > label_20240302.log 2>&1

         

        

       实时监控同步进度和状态

Nodejs内存分配太小

        

         NODE_OPTIONS='--max-old-space-size=81920'

         --scrollTime=10m

       NODE_OPTIONS='--max-old-space-size=819200' NODE_TLS_REJECT_UNAUTHORIZED=0 /home/secure/node/node-v10.24.1-linux-x64/lib/node_modules/elasticdump/bin/elasticdump --input=https://elastic:'1111111'@10.30.92.78:9200/event_20240501 --input-cert=/home/secure/ca/ca.crt --limit=100000  --ignore-errors --scrollTime=10m --output-cert=/home/secure/ca/ca7.crt --output=https://elastic:'T33333'@118.68.94.44:9200

      

     每次查询读写10w条记录 提高同步速度

迁移方案对比

         

       各个工具比较