TiDB 扩容与缩容
对于TiDB的扩容和缩容,官方文档写的非常清楚,具体参考这里 https://docs.pingcap.com/zh/tidb/stable/scale-tidb-using-tiup/,本文更多的是想基于实战的场景下对TiDB的各个节点实现扩容和缩容。
0,现有集群
现有集群如下,一共9个节点,基于上一篇的TiDB 最小拓扑架构集群安装实现。现增加一个新的节点Ubuntu25:192.168.152.135,来用这个节点实现TiDB集群中各个节点的扩容和缩容。
| 实例 | 个数 | 物理机配置 | IP | 配置 |
|---|---|---|---|---|
| TiDB | 2 | 16 VCore 32 GiB 100 GiB 用于存储 |
192.168.152.132 192.168.152.133 |
默认端口 全局目录配置 |
| PD | 3 | 4 VCore 8 GiB 100 GiB 用于存储 |
192.168.152.126 192.168.152.127 192.168.152.128 |
默认端口 全局目录配置 |
| TiKV | 3 | 16 VCore 32 GiB 2 TiB (NVMe SSD) 用于存储 |
192.168.152.129 192.168.152.130 192.168.152.131 |
默认端口 全局目录配置 |
| Monitoring & Grafana | 1 | 4 VCore 8 GiB 500 GiB (SSD) 用于存储 |
192.168.152.134 | 默认端口 全局目录配置 |

1,查看集群状态
root@ubuntu24:~# tiup cluster display TiDBCluster
Checking updates for component cluster... Timedout (after 2s)
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.152.127:2379/dashboard
Dashboard URLs: http://192.168.152.127:2379/dashboard
Grafana URL: http://192.168.152.134:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.152.134:9093 alertmanager 192.168.152.134 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
192.168.152.134:3000 grafana 192.168.152.134 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
192.168.152.126:2379 pd 192.168.152.126 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.127:2379 pd 192.168.152.127 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.128:2379 pd 192.168.152.128 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.134:9090 prometheus 192.168.152.134 9090/9115/9100/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
192.168.152.132:4000 tidb 192.168.152.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.133:4000 tidb 192.168.152.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.129:20160 tikv 192.168.152.129 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.130:20160 tikv 192.168.152.130 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.131:20160 tikv 192.168.152.131 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 11
root@ubuntu24:~#
root@ubuntu24:~#
2,ssh互信配置
配置中控机与新的目标机的ssh互信
setup_ssh_trust.sh
#!/bin/bash
# SSH trust configuration script
# ==================== Configuration ====================
HOSTS=(
"192.168.152.134"
"192.168.152.135"
)
USER="root"
PASS="123456" # 修改为实际密码
# =======================================================
# Check if trust already configured
echo ">>> Checking SSH trust status..."
all_trusted=true
for host in "${HOSTS[@]}"; do
if ssh -o ConnectTimeout=3 -o BatchMode=yes ${USER}@${host} "exit" 2>/dev/null; then
echo " ${host}: Trusted"
else
echo " ${host}: Not trusted"
all_trusted=false
fi
done
# Exit if all trusted
if $all_trusted; then
echo ">>> All hosts already trusted, skipping"
exit 0
fi
echo ">>> Starting SSH trust configuration..."
# Install sshpass
sudo apt update &>/dev/null && sudo apt install sshpass -y &>/dev/null
# Generate SSH key pair
if [ ! -f ~/.ssh/id_rsa ]; then
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa -q <<< y 2>/dev/null
fi
# Copy public key to each host
echo ">>> Copying public keys to hosts..."
for host in "${HOSTS[@]}"; do
echo " Configuring: ${host}"
sshpass -p "$PASS" ssh-copy-id -o StrictHostKeyChecking=no ${USER}@${host} &>/dev/null
done
# Collect all authorized_keys
echo ">>> Collecting authorized_keys from all hosts..."
ALL_KEYS=""
for host in "${HOSTS[@]}"; do
keys=$(ssh ${USER}@${host} 'cat ~/.ssh/authorized_keys' 2>/dev/null)
ALL_KEYS="${ALL_KEYS} ${keys}"
done
# Merge and distribute
echo ">>> Merging and distributing authorized_keys..."
echo "$ALL_KEYS" | tr ' ' '\n' | sort -u | grep -v '^$' > /tmp/all_keys
for host in "${HOSTS[@]}"; do
scp /tmp/all_keys ${USER}@${host}:~/.ssh/authorized_keys &>/dev/null
done
rm -f /tmp/all_keys
echo ">>> SSH trust configuration completed!"
3,扩容、缩容 tidb
1,tidb_server配置文件
tidb_servers:
- host: 192.168.152.135
ssh_port: 22
port: 4000
status_port: 10080
deploy_dir: /tidb-deploy/tidb-4000
log_dir: /tidb-deploy/tidb-4000/log
2,扩容检查 tiup cluster check
root@ubuntu24:/TiDB# tiup cluster check TiDBCluster /TiDB/tidb_server_scale_out.yaml --cluster
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
+ Download necessary tools
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
+ Collect basic system information
+ Collect basic system information
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... Done
- Getting system info of 192.168.152.126:22 ... Done
+ Check time zone
- Checking node 192.168.152.135 ... Done
+ Check system requirements
+ Check system requirements
+ Check system requirements
- Checking node 192.168.152.135 ... Done
- Checking node 192.168.152.135 ... Done
+ Cleanup check files
- Cleanup check files on 192.168.152.135:22 ... Done
- Cleanup check files on 192.168.152.126:22 ... Done
Node Check Result Message
---- ----- ------ -------
192.168.152.135 timezone Pass time zone is the same as the first PD machine: Etc/UTC
192.168.152.135 os-version Warn OS is Ubuntu 20.04 LTS 20.04 (Ubuntu support is not fully tested, be careful)
192.168.152.135 cpu-governor Warn unable to determine current CPU frequency governor policy
192.168.152.135 network Pass network speed of ens33 is 1000MB
192.168.152.135 thp Fail THP is enabled, please disable it for best performance
192.168.152.135 command Fail numactl not usable, bash: numactl: command not found
192.168.152.135 cpu-cores Pass number of CPU cores / threads: 2
192.168.152.135 swap Warn swap is enabled, please disable it for best performance
192.168.152.135 memory Pass memory size is 2048MB
192.168.152.135 limits Fail soft limit of 'stack' for user 'tidb' is not set or too low
192.168.152.135 limits Fail soft limit of 'nofile' for user 'tidb' is not set or too low
192.168.152.135 limits Fail hard limit of 'nofile' for user 'tidb' is not set or too low
192.168.152.135 sysctl Fail net.core.somaxconn = 4096, should 32768 or greater
192.168.152.135 sysctl Fail net.ipv4.tcp_syncookies = 1, should be 0
192.168.152.135 sysctl Fail vm.swappiness = 60, should be 0
192.168.152.135 selinux_conf Pass SELinux is disabled in configuration
192.168.152.135 selinux_status Fail executor.ssh.execute_failed: Failed to execute command over SSH for 'root@192.168.152.135:22' {ssh_stderr: bash: getenforce: command not found
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin; /usr/bin/sudo -H bash -c "getenforce"}, cause: Process exited with status 127 bash: getenforce: command not found
192.168.152.135 service Pass service firewalld not found, ignore
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
3,扩容检查修复
root@ubuntu24:/TiDB# tiup cluster check TiDBCluster /TiDB/tidb_server_scale_out.yaml --cluster --apply
Checking updates for component cluster...
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
+ Collect basic system information
+ Collect basic system information
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... Done
- Getting system info of 192.168.152.126:22 ... Done
+ Check time zone
- Checking node 192.168.152.135 ... Done
+ Check system requirements
+ Check system requirements
+ Check system requirements
- Checking node 192.168.152.135 ... Done
- Checking node 192.168.152.135 ... Done
+ Cleanup check files
- Cleanup check files on 192.168.152.135:22 ... Done
- Cleanup check files on 192.168.152.126:22 ... Done
Node Check Result Message
---- ----- ------ -------
192.168.152.135 timezone Pass time zone is the same as the first PD machine: Etc/UTC
192.168.152.135 cpu-cores Pass number of CPU cores / threads: 2
192.168.152.135 swap Warn will try to disable swap, please also check /etc/fstab manually
192.168.152.135 network Pass network speed of ens33 is 1000MB
192.168.152.135 sysctl Fail will try to set 'net.core.somaxconn = 32768'
192.168.152.135 sysctl Fail will try to set 'net.ipv4.tcp_syncookies = 0'
192.168.152.135 sysctl Fail will try to set 'vm.swappiness = 0'
192.168.152.135 selinux_conf Pass SELinux is disabled in configuration
192.168.152.135 thp Fail will try to disable THP, please check again after reboot
192.168.152.135 command Fail numactl not usable, bash: numactl: command not found, auto fixing not supported
192.168.152.135 os-version Warn OS is Ubuntu 20.04 LTS 20.04 (Ubuntu support is not fully tested, be careful), auto fixing not supported
192.168.152.135 cpu-governor Warn unable to determine current CPU frequency governor policy, auto fixing not supported
192.168.152.135 memory Pass memory size is 2048MB
192.168.152.135 limits Fail will try to set 'tidb hard nofile 1000000'
192.168.152.135 limits Fail will try to set 'tidb soft stack 10240'
192.168.152.135 limits Fail will try to set 'tidb soft nofile 1000000'
192.168.152.135 selinux_status Fail will try to disable SELinux, reboot might be needed
192.168.152.135 service Pass service firewalld not found, ignore
+ Try to apply changes to fix failed checks
+ Try to apply changes to fix failed checks
- Applying changes on 192.168.152.135 ... Error
Error: executor.ssh.execute_failed: Failed to execute command over SSH for 'root@192.168.152.135:22' {ssh_stderr: sed: can't read /etc/selinux/config: No such file or directory
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin; /usr/bin/sudo -H bash -c "sed -i 's/^[[:blank:]]*SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config && setenforce 0"}, cause: Process exited with status 2
Verbose debug logs has been written to /root/.tiup/logs/tiup-cluster-debug-2026-05-26-05-00-29.log.
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
根据你提供的错误信息,问题的核心原因是:TiUP 在执行 check --apply 时,试图修复 SELinux 配置,但目标节点 192.168.152.135 的系统中不存在 /etc/selinux/config 文件(这是 Ubuntu 系统的正常现象)。
问题分析错误直接原因
sed: can't read /etc/selinux/config: No such file or directory
TiUP 的自动修复脚本假设目标系统是 CentOS/RHEL 系列(默认使用 SELinux),执行了:
sed -i 's/^[[:blank:]]*SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
但你的节点 192.168.152.135 运行的是 Ubuntu 20.04:
Ubuntu 默认使用 AppArmor 而非 SELinux
Ubuntu 系统中不存在 /etc/selinux/config 文件
4,执行扩容命令
root@ubuntu24:/TiDB# tiup cluster scale-out TiDBCluster /TiDB/tidb_server_scale_out.yaml
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
Please confirm your topology:
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Role Host Ports OS/Arch Directories
---- ---- ----- ------- -----------
tidb 192.168.152.135 4000/10080 linux/x86_64 /tidb-deploy/tidb-4000
Attention:
1. If the topology is not what you expected, check your yaml file.
2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ Download TiDB components
- Download tidb:v8.5.6 (linux/amd64) ... Done
- Download node_exporter: (linux/amd64) ... Done
- Download blackbox_exporter: (linux/amd64) ... Done
+ Initialize target host environments
- Initialized host 192.168.152.135 ... Done
+ Deploy TiDB instance
- Deploy instance tidb -> 192.168.152.135:4000 ... Done
- Deploy node_exporter -> 192.168.152.135 ... Done
- Deploy blackbox_exporter -> 192.168.152.135 ... Done
+ Copy certificate to remote host
+ Generate scale-out config
- Generate scale-out config tidb -> 192.168.152.135:4000 ... Done
+ Init monitor config
- Generate config node_exporter -> 192.168.152.135 ... Done
- Generate config blackbox_exporter -> 192.168.152.135 ... Done
Enabling component tidb
Enabling instance 192.168.152.135:4000
Enable instance 192.168.152.135:4000 success
Enabling component node_exporter
Enabling instance 192.168.152.135
Enable 192.168.152.135 success
Enabling component blackbox_exporter
Enabling instance 192.168.152.135
Enable 192.168.152.135 success
+ [ Serial ] - Save meta
+ [ Serial ] - Start new instances
Starting component tidb
Starting instance 192.168.152.135:4000
Start instance 192.168.152.135:4000 success
Starting component node_exporter
Starting instance 192.168.152.135
Start 192.168.152.135 success
Starting component blackbox_exporter
Starting instance 192.168.152.135
Start 192.168.152.135 success
+ Refresh components conifgs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh components conifgs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config tidb -> 192.168.152.135:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
Scaled cluster `TiDBCluster` out successfully
root@ubuntu24:/TiDB#

执行扩容命令执行,在中转机上执行tiup cluster enable TiDBCluster来生成systemctl 服务
5,缩容tidb_servers
如果是所用TiKV,类似于上面但是需要修改端口号tiup cluster scale-in TiDBCluster --node ***.***.***.***.20160,这个端口号是集群节点对外通讯的端口号
移除 TiDB、PD 节点和移除 TiKV 节点的步骤类似。
由于 TiKV 和 TiFlash 组件是异步下线的,且下线过程耗时较长,所以 TiUP 对 TiKV 和 TiFlash 组件做了特殊处理,详情参考下线特殊处理。
TiKV 中的 PD Client 会缓存 PD 节点的列表。当前版本的 TiKV 有定期自动更新 PD 节点的机制,可以降低 TiKV 缓存的 PD 节点列表过旧这一问题出现的概率。但你应尽量避免在扩容新 PD 后直接一次性缩容所有扩容前就已经存在的 PD 节点。如果需要,请确保在下线所有之前存在的 PD 节点前将 PD 的 leader 切换至新扩容的 PD 节点。
root@ubuntu24:/TiDB# tiup cluster scale-in TiDBCluster --node 192.168.152.135:4000
This operation will delete the 192.168.152.135:4000 nodes in `TiDBCluster` and all their data.
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[192.168.152.135:4000] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:root SSHProxyIdentity:/root/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 SSHCustomScripts:{BeforeRestartInstance:{Raw:} AfterRestartInstance:{Raw:}} CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}
Stopping component tidb
Stopping instance 192.168.152.135
Stop tidb 192.168.152.135:4000 success
Destroying component tidb
Destroying instance 192.168.152.135
Destroy 192.168.152.135 finished
- Destroy tidb paths: [/tidb-deploy/tidb-4000/log /tidb-deploy/tidb-4000 /etc/systemd/system/tidb-4000.service]
Stopping component node_exporter
Stopping instance 192.168.152.135
Stop 192.168.152.135 success
Stopping component blackbox_exporter
Stopping instance 192.168.152.135
Stop 192.168.152.135 success
Destroying monitored 192.168.152.135
Destroying instance 192.168.152.135
Destroy monitored on 192.168.152.135 success
Delete public key 192.168.152.135
Delete public key 192.168.152.135 success
+ [ Serial ] - UpdateMeta: cluster=TiDBCluster, deleted=`'192.168.152.135:4000'`
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
Scaled cluster `TiDBCluster` in successfully
root@ubuntu24:/TiDB#
4,扩容,缩容 TiKV
1,tikv配置文件
tikv_servers:
- host: 192.168.152.135
ssh_port: 22
port: 20160
status_port: 20180
deploy_dir: /tidb-deploy/tikv-20160
data_dir: /tidb-data/tikv-20160
log_dir: /tidb-deploy/tikv-20160/log
2,扩容检查
root@ubuntu24:/TiDB# tiup cluster check TiDBCluster /TiDB/tikv_server_scale_out.yaml --cluster
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
+ Collect basic system information
+ Collect basic system information
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... Done
- Getting system info of 192.168.152.126:22 ... Done
+ Check time zone
- Checking node 192.168.152.135 ... Done
+ Check system requirements
+ Check system requirements
+ Check system requirements
- Checking node 192.168.152.135 ... Done
- Checking node 192.168.152.135 ... Done
+ Cleanup check files
- Cleanup check files on 192.168.152.135:22 ... Done
- Cleanup check files on 192.168.152.126:22 ... Done
Node Check Result Message
---- ----- ------ -------
192.168.152.135 service Pass service firewalld not found, ignore
192.168.152.135 command Fail numactl not usable, bash: numactl: command not found
192.168.152.135 os-version Warn OS is Ubuntu 20.04 LTS 20.04 (Ubuntu support is not fully tested, be careful)
192.168.152.135 network Pass network speed of ens33 is 1000MB
192.168.152.135 disk Fail mount point / does not have 'nodelalloc' option set
192.168.152.135 disk Warn mount point / does not have 'noatime' option set
192.168.152.135 selinux_conf Pass SELinux is disabled in configuration
192.168.152.135 thp Fail THP is enabled, please disable it for best performance
192.168.152.135 timezone Pass time zone is the same as the first PD machine: Etc/UTC
192.168.152.135 cpu-cores Pass number of CPU cores / threads: 2
192.168.152.135 cpu-governor Warn unable to determine current CPU frequency governor policy
192.168.152.135 memory Pass memory size is 2048MB
192.168.152.135 selinux_status Fail executor.ssh.execute_failed: Failed to execute command over SSH for 'root@192.168.152.135:22' {ssh_stderr: bash: getenforce: command not found
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin; /usr/bin/sudo -H bash -c "getenforce"}, cause: Process exited with status 127 bash: getenforce: command not found
root@ubuntu24:/TiDB#
3,扩容检查修改
root@ubuntu24:/TiDB# tiup cluster check TiDBCluster /TiDB/tikv_server_scale_out.yaml --cluster --apply
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
+ Collect basic system information
+ Collect basic system information
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... Done
- Getting system info of 192.168.152.126:22 ... Done
+ Check time zone
- Checking node 192.168.152.135 ... Done
+ Check system requirements
+ Check system requirements
+ Check system requirements
- Checking node 192.168.152.135 ... Done
- Checking node 192.168.152.135 ... Done
+ Cleanup check files
- Cleanup check files on 192.168.152.135:22 ... Done
- Cleanup check files on 192.168.152.126:22 ... Done
Node Check Result Message
---- ----- ------ -------
192.168.152.135 cpu-governor Warn unable to determine current CPU frequency governor policy, auto fixing not supported
192.168.152.135 memory Pass memory size is 2048MB
192.168.152.135 disk Warn mount point / does not have 'noatime' option set, auto fixing not supported
192.168.152.135 selinux_conf Pass SELinux is disabled in configuration
192.168.152.135 timezone Pass time zone is the same as the first PD machine: Etc/UTC
192.168.152.135 network Pass network speed of ens33 is 1000MB
192.168.152.135 disk Fail mount point / does not have 'nodelalloc' option set, auto fixing not supported
192.168.152.135 selinux_status Fail will try to disable SELinux, reboot might be needed
192.168.152.135 thp Fail will try to disable THP, please check again after reboot
192.168.152.135 service Pass service firewalld not found, ignore
192.168.152.135 command Fail numactl not usable, bash: numactl: command not found, auto fixing not supported
192.168.152.135 os-version Warn OS is Ubuntu 20.04 LTS 20.04 (Ubuntu support is not fully tested, be careful), auto fixing not supported
192.168.152.135 cpu-cores Pass number of CPU cores / threads: 2
+ Try to apply changes to fix failed checks
- Applying changes on 192.168.152.135 ... Error
Error: executor.ssh.execute_failed: Failed to execute command over SSH for 'root@192.168.152.135:22' {ssh_stderr: sed: can't read /etc/selinux/config: No such file or directory
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin; /usr/bin/sudo -H bash -c "sed -i 's/^[[:blank:]]*SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config && setenforce 0"}, cause: Process exited with status 2
Verbose debug logs has been written to /root/.tiup/logs/tiup-cluster-debug-2026-05-26-05-30-09.log.
root@ubuntu24:/TiDB#
4,执行扩容
root@ubuntu24:/TiDB# tiup cluster scale-out TiDBCluster /TiDB/tikv_server_scale_out.yaml
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
Please confirm your topology:
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Role Host Ports OS/Arch Directories
---- ---- ----- ------- -----------
tikv 192.168.152.135 20160/20180 linux/x86_64 /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
Attention:
1. If the topology is not what you expected, check your yaml file.
2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ Download TiDB components
- Download tikv:v8.5.6 (linux/amd64) ... Done
- Download node_exporter: (linux/amd64) ... Done
- Download blackbox_exporter: (linux/amd64) ... Done
+ Initialize target host environments
- Initialized host 192.168.152.135 ... Done
+ Deploy TiDB instance
- Deploy instance tikv -> 192.168.152.135:20160 ... Done
- Deploy node_exporter -> 192.168.152.135 ... Done
- Deploy blackbox_exporter -> 192.168.152.135 ... Done
+ Copy certificate to remote host
+ Generate scale-out config
- Generate scale-out config tikv -> 192.168.152.135:20160 ... Done
+ Init monitor config
- Generate config node_exporter -> 192.168.152.135 ... Done
- Generate config blackbox_exporter -> 192.168.152.135 ... Done
Enabling component tikv
Enabling instance 192.168.152.135:20160
Enable instance 192.168.152.135:20160 success
Enabling component node_exporter
Enabling instance 192.168.152.135
Enable 192.168.152.135 success
Enabling component blackbox_exporter
Enabling instance 192.168.152.135
Enable 192.168.152.135 success
+ [ Serial ] - Save meta
+ [ Serial ] - Start new instances
Starting component tikv
Starting instance 192.168.152.135:20160
Start instance 192.168.152.135:20160 success
Starting component node_exporter
Starting instance 192.168.152.135
Start 192.168.152.135 success
Starting component blackbox_exporter
Starting instance 192.168.152.135
Start 192.168.152.135 success
+ Refresh components conifgs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh components conifgs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tikv -> 192.168.152.135:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
Scaled cluster `TiDBCluster` out successfully
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB# tiup cluster display TiDBCluster
Checking updates for component cluster... Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.152.127:2379/dashboard
Dashboard URLs: http://192.168.152.127:2379/dashboard
Grafana URL: http://192.168.152.134:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.152.134:9093 alertmanager 192.168.152.134 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
192.168.152.134:3000 grafana 192.168.152.134 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
192.168.152.126:2379 pd 192.168.152.126 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.127:2379 pd 192.168.152.127 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.128:2379 pd 192.168.152.128 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.134:9090 prometheus 192.168.152.134 9090/9115/9100/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
192.168.152.132:4000 tidb 192.168.152.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.133:4000 tidb 192.168.152.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.129:20160 tikv 192.168.152.129 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.130:20160 tikv 192.168.152.130 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.131:20160 tikv 192.168.152.131 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.135:20160 tikv 192.168.152.135 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 12
root@ubuntu24:/TiDB#
5,缩容TiKV
root@ubuntu24:/TiDB# tiup cluster scale-in TiDBCluster --node 192.168.152.135:20160
This operation will delete the 192.168.152.135:20160 nodes in `TiDBCluster` and all their data.
Do you want to continue? [y/N]:(default=N) y
The component `[tikv]` will become tombstone, maybe exists in several minutes or hours, after that you can use the prune command to clean it
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[192.168.152.135:20160] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:root SSHProxyIdentity:/root/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 SSHCustomScripts:{BeforeRestartInstance:{Raw:} AfterRestartInstance:{Raw:}} CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}
The component `tikv` will become tombstone, maybe exists in several minutes or hours, after that you can use the prune command to clean it
+ [ Serial ] - UpdateMeta: cluster=TiDBCluster, deleted=`''`
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
Scaled cluster `TiDBCluster` in successfully
root@ubuntu24:/TiDB#
5,扩容,缩容 PD
1,pd配置文件
pd_servers:
- host: 192.168.152.135
ssh_port: 22
name: 192.168.152.135:2379
client_port: 2379
peer_port: 2380
deploy_dir: /tidb-deploy/pd-2379
data_dir: /tidb-data/pd-2379
log_dir: /tidb-deploy/pd-2379/log
2,扩容检查
命令:tiup cluster check TiDBCluster /TiDB/pd_server_scale_out.yaml --cluster
root@ubuntu24:/TiDB# tiup cluster check TiDBCluster /TiDB/pd_server_scale_out.yaml --cluster
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
+ Collect basic system information
+ Collect basic system information
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... Done
- Getting system info of 192.168.152.126:22 ... Done
+ Check time zone
- Checking node 192.168.152.135 ... Done
+ Check system requirements
+ Check system requirements
+ Check system requirements
- Checking node 192.168.152.135 ... Done
- Checking node 192.168.152.135 ... Done
+ Cleanup check files
- Cleanup check files on 192.168.152.135:22 ... Done
- Cleanup check files on 192.168.152.126:22 ... Done
Node Check Result Message
---- ----- ------ -------
192.168.152.135 selinux_status Fail executor.ssh.execute_failed: Failed to execute command over SSH for 'root@192.168.152.135:22' {ssh_stderr: bash: getenforce: command not found
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin; /usr/bin/sudo -H bash -c "getenforce"}, cause: Process exited with status 127 bash: getenforce: command not found
192.168.152.135 memory Pass memory size is 2048MB
192.168.152.135 disk Warn mount point / does not have 'noatime' option set
192.168.152.135 thp Fail THP is enabled, please disable it for best performance
192.168.152.135 service Pass service firewalld not found, ignore
192.168.152.135 command Fail numactl not usable, bash: numactl: command not found
192.168.152.135 timezone Pass time zone is the same as the first PD machine: Etc/UTC
192.168.152.135 os-version Warn OS is Ubuntu 20.04 LTS 20.04 (Ubuntu support is not fully tested, be careful)
192.168.152.135 cpu-cores Pass number of CPU cores / threads: 2
192.168.152.135 cpu-governor Warn unable to determine current CPU frequency governor policy
192.168.152.135 network Pass network speed of ens33 is 1000MB
192.168.152.135 disk Fail mount point / does not have 'nodelalloc' option set
192.168.152.135 selinux_conf Pass SELinux is disabled in configuration
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
3,扩容检查修改
命令:tiup cluster check TiDBCluster /TiDB/pd_server_scale_out.yaml --cluster --apply
root@ubuntu24:/TiDB# tiup cluster check TiDBCluster /TiDB/pd_server_scale_out.yaml --cluster --apply
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... ⠇ CopyComponent: component=insight, version=, remote=192.168.152.135:/tmp/tiup os=linux, arch=amd64
+ Collect basic system information
- Getting system info of 192.168.152.135:22 ... Done
- Getting system info of 192.168.152.126:22 ... Done
+ Check time zone
- Checking node 192.168.152.135 ... Done
+ Check system requirements
+ Check system requirements
+ Check system requirements
- Checking node 192.168.152.135 ... Done
- Checking node 192.168.152.135 ... Done
+ Cleanup check files
- Cleanup check files on 192.168.152.135:22 ... Done
- Cleanup check files on 192.168.152.126:22 ... Done
Node Check Result Message
---- ----- ------ -------
192.168.152.135 selinux_status Fail will try to disable SELinux, reboot might be needed
192.168.152.135 command Fail numactl not usable, bash: numactl: command not found, auto fixing not supported
192.168.152.135 os-version Warn OS is Ubuntu 20.04 LTS 20.04 (Ubuntu support is not fully tested, be careful), auto fixing not supported
192.168.152.135 disk Fail mount point / does not have 'nodelalloc' option set, auto fixing not supported
192.168.152.135 disk Warn mount point / does not have 'noatime' option set, auto fixing not supported
192.168.152.135 thp Fail will try to disable THP, please check again after reboot
192.168.152.135 service Pass service firewalld not found, ignore
192.168.152.135 timezone Pass time zone is the same as the first PD machine: Etc/UTC
192.168.152.135 cpu-cores Pass number of CPU cores / threads: 2
192.168.152.135 cpu-governor Warn unable to determine current CPU frequency governor policy, auto fixing not supported
192.168.152.135 memory Pass memory size is 2048MB
192.168.152.135 network Pass network speed of ens33 is 1000MB
192.168.152.135 selinux_conf Pass SELinux is disabled in configuration
+ Try to apply changes to fix failed checks
- Applying changes on 192.168.152.135 ... Error
Error: executor.ssh.execute_failed: Failed to execute command over SSH for 'root@192.168.152.135:22' {ssh_stderr: sed: can't read /etc/selinux/config: No such file or directory
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin; /usr/bin/sudo -H bash -c "sed -i 's/^[[:blank:]]*SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config && setenforce 0"}, cause: Process exited with status 2
Verbose debug logs has been written to /root/.tiup/logs/tiup-cluster-debug-2026-05-26-06-27-36.log.
root@ubuntu24:/TiDB#
4,执行扩容
扩容命令:tiup cluster scale-out TiDBCluster /TiDB/pd_server_scale_out.yaml
可以发现扩容成功并成功加入集群
root@ubuntu24:/TiDB# tiup cluster scale-out TiDBCluster /TiDB/pd_server_scale_out.yaml
+ Detect CPU Arch Name
- Detecting node 192.168.152.135 Arch info ... Done
+ Detect CPU OS Name
- Detecting node 192.168.152.135 OS info ... Done
Please confirm your topology:
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Role Host Ports OS/Arch Directories
---- ---- ----- ------- -----------
pd 192.168.152.135 2379/2380 linux/x86_64 /tidb-deploy/pd-2379,/tidb-data/pd-2379
Attention:
1. If the topology is not what you expected, check your yaml file.
2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ Download TiDB components
- Download pd:v8.5.6 (linux/amd64) ... Done
+ Initialize target host environments
+ Deploy TiDB instance
- Deploy instance pd -> 192.168.152.135:2379 ... Done
+ Copy certificate to remote host
+ Generate scale-out config
- Generate scale-out config pd -> 192.168.152.135:2379 ... Done
+ Init monitor config
Enabling component pd
Enabling instance 192.168.152.135:2379
Enable instance 192.168.152.135:2379 success
Enabling component node_exporter
Enabling instance 192.168.152.135
Enable 192.168.152.135 success
Enabling component blackbox_exporter
Enabling instance 192.168.152.135
Enable 192.168.152.135 success
+ [ Serial ] - Save meta
+ [ Serial ] - Start new instances
Starting component pd
Starting instance 192.168.152.135:2379
Start instance 192.168.152.135:2379 success
Starting component node_exporter
Starting instance 192.168.152.135
Start 192.168.152.135 success
Starting component blackbox_exporter
Starting instance 192.168.152.135
Start 192.168.152.135 success
+ Refresh components conifgs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh components conifgs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config pd -> 192.168.152.135:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tikv -> 192.168.152.135:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
Scaled cluster `TiDBCluster` out successfully
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB# tiup cluster display TiDBCluster
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.152.127:2379/dashboard
Dashboard URLs: http://192.168.152.127:2379/dashboard
Grafana URL: http://192.168.152.134:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.152.134:9093 alertmanager 192.168.152.134 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
192.168.152.134:3000 grafana 192.168.152.134 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
192.168.152.126:2379 pd 192.168.152.126 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.127:2379 pd 192.168.152.127 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.128:2379 pd 192.168.152.128 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.135:2379 pd 192.168.152.135 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.134:9090 prometheus 192.168.152.134 9090/9115/9100/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
192.168.152.132:4000 tidb 192.168.152.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.133:4000 tidb 192.168.152.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.129:20160 tikv 192.168.152.129 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.130:20160 tikv 192.168.152.130 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.131:20160 tikv 192.168.152.131 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.135:20160 tikv 192.168.152.135 20160/20180 linux/x86_64 Tombstone /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 13
Total nodes: 13
There are some nodes can be pruned:
Nodes: [192.168.152.135:20160]
You can destroy them with the command: `tiup cluster prune TiDBCluster`
root@ubuntu24:/TiDB#
这里发现192.168.152.135 20160/20180 linux/x86_64 处于Tombstone,因为之前对TiKV缩容,这个节点下线之后并没有彻底清理,,根据系统的提示tiup cluster prune TiDBCluster,清理集群中所有状态为 Tombstone 的实例。
root@ubuntu24:/TiDB# tiup cluster display TiDBCluster
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.152.127:2379/dashboard
Dashboard URLs: http://192.168.152.127:2379/dashboard
Grafana URL: http://192.168.152.134:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.152.134:9093 alertmanager 192.168.152.134 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
192.168.152.134:3000 grafana 192.168.152.134 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
192.168.152.126:2379 pd 192.168.152.126 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.127:2379 pd 192.168.152.127 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.128:2379 pd 192.168.152.128 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.135:2379 pd 192.168.152.135 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.134:9090 prometheus 192.168.152.134 9090/9115/9100/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
192.168.152.132:4000 tidb 192.168.152.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.133:4000 tidb 192.168.152.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.129:20160 tikv 192.168.152.129 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.130:20160 tikv 192.168.152.130 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.131:20160 tikv 192.168.152.131 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.135:20160 tikv 192.168.152.135 20160/20180 linux/x86_64 Tombstone /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 13
There are some nodes can be pruned:
Nodes: [192.168.152.135:20160]
You can destroy them with the command: `tiup cluster prune TiDBCluster`
root@ubuntu24:/TiDB#
root@ubuntu24:/TiDB# tiup cluster prune TiDBCluster
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ [ Serial ] - FindTomestoneNodes
Will destroy these nodes: [192.168.152.135:20160]
Do you confirm this action? [y/N]:(default=N) y
Start destroy Tombstone nodes: [192.168.152.135:20160] ...
+ [ Serial ] - ClusterOperate: operation=ScaleInOperation, options={Roles:[] Nodes:[] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:root SSHProxyIdentity:/root/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 SSHCustomScripts:{BeforeRestartInstance:{Raw:} AfterRestartInstance:{Raw:}} CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}
Stopping component tikv
Stopping instance 192.168.152.135
Stop tikv 192.168.152.135:20160 success
Destroying component tikv
Destroying instance 192.168.152.135
Destroy 192.168.152.135 finished
- Destroy tikv paths: [/tidb-data/tikv-20160 /tidb-deploy/tikv-20160/log /tidb-deploy/tikv-20160 /etc/systemd/system/tikv-20160.service]
+ [ Serial ] - UpdateMeta: cluster=TiDBCluster, deleted=`'192.168.152.135:20160'`
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config pd -> 192.168.152.135:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
+ [ Serial ] - RemoveTomestoneNodesInPD
Destroy success
root@ubuntu24:/TiDB# tiup cluster display TiDBCluster
Cluster type: tidb
Cluster name: TiDBCluster
Cluster version: v8.5.6
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.152.127:2379/dashboard
Dashboard URLs: http://192.168.152.127:2379/dashboard
Grafana URL: http://192.168.152.134:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.152.134:9093 alertmanager 192.168.152.134 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
192.168.152.134:3000 grafana 192.168.152.134 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
192.168.152.126:2379 pd 192.168.152.126 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.127:2379 pd 192.168.152.127 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.128:2379 pd 192.168.152.128 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.135:2379 pd 192.168.152.135 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.152.134:9090 prometheus 192.168.152.134 9090/9115/9100/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
192.168.152.132:4000 tidb 192.168.152.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.133:4000 tidb 192.168.152.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.152.129:20160 tikv 192.168.152.129 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.130:20160 tikv 192.168.152.130 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.152.131:20160 tikv 192.168.152.131 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 12

5,缩容pd
缩容命令:tiup cluster scale-in TiDBCluster --node 192.168.152.135:2379
root@ubuntu24:/TiDB# tiup cluster scale-in TiDBCluster --node 192.168.152.135:2379
This operation will delete the 192.168.152.135:2379 nodes in `TiDBCluster` and all their data.
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.135
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[192.168.152.135:2379] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:root SSHProxyIdentity:/root/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 SSHCustomScripts:{BeforeRestartInstance:{Raw:} AfterRestartInstance:{Raw:}} CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}
Stopping component pd
Stopping instance 192.168.152.135
Stop pd 192.168.152.135:2379 success
Destroying component pd
Destroying instance 192.168.152.135
Destroy 192.168.152.135 finished
- Destroy pd paths: [/tidb-data/pd-2379 /tidb-deploy/pd-2379/log /tidb-deploy/pd-2379 /etc/systemd/system/pd-2379.service]
Stopping component node_exporter
Stopping instance 192.168.152.135
Stop 192.168.152.135 success
Stopping component blackbox_exporter
Stopping instance 192.168.152.135
Stop 192.168.152.135 success
Destroying monitored 192.168.152.135
Destroying instance 192.168.152.135
Destroy monitored on 192.168.152.135 success
Delete public key 192.168.152.135
Delete public key 192.168.152.135 success
+ [ Serial ] - UpdateMeta: cluster=TiDBCluster, deleted=`'192.168.152.135:2379'`
+ [ Serial ] - UpdateTopology: cluster=TiDBCluster
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/TiDBCluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.130
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.131
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.127
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.133
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.134
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.126
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.128
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.129
+ [Parallel] - UserSSH: user=tidb, host=192.168.152.132
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
+ Refresh instance configs
- Generate config pd -> 192.168.152.126:2379 ... Done
- Generate config pd -> 192.168.152.127:2379 ... Done
- Generate config pd -> 192.168.152.128:2379 ... Done
- Generate config tikv -> 192.168.152.129:20160 ... Done
- Generate config tikv -> 192.168.152.130:20160 ... Done
- Generate config tikv -> 192.168.152.131:20160 ... Done
- Generate config tidb -> 192.168.152.132:4000 ... Done
- Generate config tidb -> 192.168.152.133:4000 ... Done
- Generate config prometheus -> 192.168.152.134:9090 ... Done
- Generate config grafana -> 192.168.152.134:3000 ... Done
- Generate config alertmanager -> 192.168.152.134:9093 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 192.168.152.134:9090 ... Done
- Reload grafana -> 192.168.152.134:3000 ... Done
Scaled cluster `TiDBCluster` in successfully
root@ubuntu24:/TiDB#
20260527备注:应该是上头了,昨晚上做梦都梦见扩容TiKV异常的问题,梦里也是各种排查,最近比较忙,有时间再重新测试tikv扩容的问题。
浙公网安备 33010602011771号