Redis Cluster (Redis集群) 方案-2014

Redis Cluster (Redis集群) 方案研究
20140620 Chenxin
参考资料:
官网集群介绍
http://redis.readthedocs.org/en/latest/topic/cluster-tutorial.html
http://redis.readthedocs.org/en/latest/topic/cluster-spec.html#cluster-spec
英文官网:http://redis.io
中文官网:http://www.redis.cn
当前redis的cluster还处于测试阶段,官方已经发布redis3.0.0(第5个beta测试版)(2.9.54).

1.安装
wget https://github.com/antirez/redis/archive/3.0.0-beta5.tar.gz
tar xvf 3.0.0-beta5.tar.gz
cd redis-3.0.0-beta5/
mkdir /usr/local/redis-3.0.0/
make PREFIX=/usr/local/redis-3.0.0/ install #只编译生成可执行的二进制文件
cd /usr/local/redis-3.0.0/
mkdir cluster-test
cd cluster-test/
mkdir 7000 7001 7002 7003 7004 7005

2.配置文件
源码解压后的参考配置文件,如下:
cat redis.conf |grep -v -E "(#|^$)"
daemonize no
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

这里以龙之力量的配置文件作为参考
cd 7000
vim redis.conf
dir /usr/local/redis-3.0.0/cluster-test/7000/
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

requirepass w09cim32srpghj

bind 192.168.1.170

timeout 0

pidfile ./run/redis.pid
logfile ./logs/stdout.log
dbfilename dump.rdb #3.0版本里的数据文件不支持目录
daemonize yes
databases 16
loglevel verbose
save 900 1
save 300 10
save 60 10000
rdbcompression yes
slave-serve-stale-data yes
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
activerehashing yes

已下参数均不在生效,且支持方式有变

vm-enabled no

vm-swap-file /tmp/redis.swap

vm-max-memory 0

vm-page-size 32

vm-pages 134217728

vm-max-threads 4

hash-max-zipmap-entries 512

hash-max-zipmap-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

3.启动进程
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7000/redis.conf
其他的7001-7005这几个的启动:
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7001/redis.conf
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7002/redis.conf
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7003/redis.conf
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7004/redis.conf
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7005/redis.conf

启动日志如下:
25498:M 25 Jun 10:47:33.261 * The server is now ready to accept connections on port 7000
25498:M 25 Jun 10:47:34.259 - 0 clients connected (0 slaves), 1158008 bytes in use
25498:M 25 Jun 10:47:39.270 - 0 clients connected (0 slaves), 1158008 bytes in use
...

生成文件:
ls /usr/local/redis-3.0.0/cluster-test/7000/
appendonly.aof dump.rdb(这个数据文件会在集群建立后自动生成) logs/ nodes.conf redis.conf run/

4.创建集群
redis-trib.rb程序通过向实例发送特殊命令来完成创建新集群,检查集群,或者对集群进行重新分片(reshared)等工作.
cp -aprf /opt/redis-3.0.0-beta5/src/redis-trib.rb /usr/local/redis-3.0.0/bin/
cd /usr/local/redis-3.0.0/bin/
./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
这一步会报很多错误,解决方法如下:
yum install ruby rubygems ruby-devel rubygems-devel -y
安装ruby访问redis的插件:
gem install redis
重新执行 ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

Creating cluster
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7001: OK
Connecting to node 127.0.0.1:7002: OK
Connecting to node 127.0.0.1:7003: OK
Connecting to node 127.0.0.1:7004: OK
Connecting to node 127.0.0.1:7005: OK
Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: 6cc6ae11e0f56bea61b8c6436e78dd09581d2438 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: 828bf05ddb9d343ec5762c0882de405b3df8c7af 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 9f353461f6168ae10c67c057a9acfa58a8849bf5 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
S: ffb55461a26499944fd95823dc9dd9f51b258c4a 127.0.0.1:7003
replicates 6cc6ae11e0f56bea61b8c6436e78dd09581d2438
S: 25d29b47ae2516454b3d8f494f1527bb773c8d05 127.0.0.1:7004
replicates 828bf05ddb9d343ec5762c0882de405b3df8c7af
S: cb9e2ac33766ec7c6d203965d6f8ed6d889c0b70 127.0.0.1:7005
replicates 9f353461f6168ae10c67c057a9acfa58a8849bf5
Can I set the above configuration? (type 'yes' to accept): yes
Nodes configuration updated
Assign a different config epoch to each node
Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
Performing Cluster Check (using node 127.0.0.1:7000)
M: 6cc6ae11e0f56bea61b8c6436e78dd09581d2438 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: 828bf05ddb9d343ec5762c0882de405b3df8c7af 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 9f353461f6168ae10c67c057a9acfa58a8849bf5 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
M: ffb55461a26499944fd95823dc9dd9f51b258c4a 127.0.0.1:7003
slots: (0 slots) master
replicates 6cc6ae11e0f56bea61b8c6436e78dd09581d2438
M: 25d29b47ae2516454b3d8f494f1527bb773c8d05 127.0.0.1:7004
slots: (0 slots) master
replicates 828bf05ddb9d343ec5762c0882de405b3df8c7af
M: cb9e2ac33766ec7c6d203965d6f8ed6d889c0b70 127.0.0.1:7005
slots: (0 slots) master
replicates 9f353461f6168ae10c67c057a9acfa58a8849bf5
[OK] All nodes agree about slots configuration.
Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.

附:redis-trib.rb的参数
原脚本文件:
"create" => ["create_cluster_cmd", -2, "host1:port1 ... hostN:portN"],
"check" => ["check_cluster_cmd", 2, "host:port"],
"fix" => ["fix_cluster_cmd", 2, "host:port"],
"reshard" => ["reshard_cluster_cmd", 2, "host:port"],
"add-node" => ["addnode_cluster_cmd", 3, "new_host:new_port existing_host:existing_port"],
"del-node" => ["delnode_cluster_cmd", 3, "host:port node_id"],
"set-timeout" => ["set_timeout_cluster_cmd", 3, "host:port milliseconds"],
"call" => ["call_cluster_cmd", -3, "host:port command arg arg .. arg"],
"import" => ["import_cluster_cmd", 2, "host:port"],
"help" => ["help_cluster_cmd", 1, "(show this help)"]
帮助说明:
/usr/local/redis-3.0.0/bin/redis-trib.rb help
Usage: redis-trib <arguments ...>
help (show this help)
del-node host:port node_id
reshard host:port
call host:port command arg arg .. arg
add-node new_host:new_port existing_host:existing_port
--master-id
--slave
create host1:port1 ... hostN:portN
--replicas
import host:port
--from
fix host:port
set-timeout host:port milliseconds
check host:port
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

5.检查集群
检查各个redis启动后的logs/stdout.log日志文件是否正常.类似:
tail logs/stdout.log
25498:M 25 Jun 11:26:18.940 - 0 clients connected (1 slaves), 2322624 bytes in use
25498:M 25 Jun 11:26:23.950 - 0 clients connected (1 slaves), 2322624 bytes in use
25498:M 25 Jun 11:26:28.960 - 0 clients connected (1 slaves), 2322624 bytes in use
/usr/local/redis-3.0.0/bin/redis-cli -p 7000

cluster info
...
cluster nodes使用说明
redis-cli -p 7000 cluster nodes
3fc783611028b1707fd65345e763befb36454d73 127.0.0.1:7004 slave 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 0 1385503418521 0 connected
a211e242fc6b22a9427fed61285e85892fa04e08 127.0.0.1:7003 slave 97a3a64667477371c4479320d683e4c8db5858b1 0 1385503419023 0 connected
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5959 10922-11422
3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7005 master - 0 1385503419023 3 connected 11423-16383
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385503417005 0 connected 5960-10921
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385503418016 3 connected
节点ID
ip:port 节点的IP地址和端口号.例如 127.0.0.1:7000,其中 :0 表示的是客户端当前连接着的IP地址和端口号.
flags 节点的角色(如 master,slave ,myself)以及状态(例如 fail ,等等).
如果节点是一个从节点的话, 那么跟在 flags 之后的将是主节点的节点 ID.
0集群最近一次向节点发送 PING 命令之后,过去了多长时间还没接到回复.
1385503418521节点最近一次返回 PONG 回复的时间.
0节点的配置纪元(configuration epoch)：详细信息请参考 Redis 集群规范 .
connected本节点的网络连接情况：例如 connected .
10922-11422节点目前包含的槽:例如 127.0.0.1:7001.

6.向集群写入或读取数据
相对于客户端来说(应用程序),master和slave地位相当,都可以"保存"或读取
[root@lg-2-2-1-n23-84 cluster-test]# /usr/local/redis-3.0.0/bin/redis-cli -c -p 7000
127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002 #把数据根据槽位直接定向到7002(master2)里去保存了.
OK
127.0.0.1:7002> get foo
"bar"
[root@lg-2-2-1-n23-84 cluster-test]# /usr/local/redis-3.0.0/bin/redis-cli -c -p 7001
127.0.0.1:7001> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7002
"bar"
[root@lg-2-2-1-n23-84 cluster-test]# /usr/local/redis-3.0.0/bin/redis-cli -c -p 7005
127.0.0.1:7005> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7002
"bar"
[root@lg-2-2-1-n23-84 cluster-test]# /usr/local/redis-3.0.0/bin/redis-cli -c -p 7005
127.0.0.1:7005> set foo2 bar2
-> Redirected to slot [1044] located at 127.0.0.1:7000
OK
127.0.0.1:7000> get foo2
"bar2"

7.关闭集群
/usr/local/redis-3.0.0/bin/redis-cli -p 7005 shutdown
/usr/local/redis-3.0.0/bin/redis-cli -p 7004 shutdown
...关闭集群最好是先关闭slave,然后逐个关闭master.通过node.conf文件可以看到哪些是master,哪些是slave.
在关闭过程中,所有运行着的redis进程都会报错,且集群里状态是失败状态.类似如下:
25582:S 25 Jun 11:31:52.050 # Error condition on socket for SYNC: Connection refused
25582:S 25 Jun 11:31:53.055 * Connecting to MASTER 127.0.0.1:7002
25582:S 25 Jun 11:31:53.056 * MASTER <-> SLAVE sync started
25582:S 25 Jun 11:31:53.056 # Error condition on socket for SYNC: Connection refused
25582:S 25 Jun 11:31:54.061 - 0 clients connected (0 slaves), 1228376 bytes in use
25582:S 25 Jun 11:31:54.061 * Connecting to MASTER 127.0.0.1:7002
25582:S 25 Jun 11:31:54.061 * MASTER <-> SLAVE sync started
25582:S 25 Jun 11:31:54.062 # Error condition on socket for SYNC: Connection refused
25582:S 25 Jun 11:31:54.161 - Accepted 127.0.0.1:51327
25582:S 25 Jun 11:31:54.161 # User requested shutdown...
25582:S 25 Jun 11:31:54.161 * Calling fsync() on the AOF file.
25582:S 25 Jun 11:31:54.161 * Saving the final RDB snapshot before exiting.
25582:S 25 Jun 11:31:54.176 * DB saved on disk
25582:S 25 Jun 11:31:54.176 * Removing the pid file.
25582:S 25 Jun 11:31:54.176 # Redis is now ready to exit, bye bye...
启动集群的顺序按照先主后从的顺序.

8.添加节点
如果要添加的新节点是一个主节点,那么我们需要创建一个空节点（empty node）,然后将某些哈希桶移动到这个空节点里面.
如果要添加的新节点是一个从节点,那么我们需要将这个新节点设置为集群中某个节点的复制品（replica）.

无论添加的是那种节点,第一步要做的总是添加一个空节点.

我们可以继续使用之前启动 127.0.0.1:7000,127.0.0.1:7001 等节点的方法,创建一个端口号为7006的新节点,使用的配置文件也和之前一样,只是记得要将配置中的端口号改为7006.

以下是启动端口号为 7006 的新节点的详细步骤：
进入 cluster-test 文件夹.创建并进入 7006 文件夹.将 redis.conf 文件复制到 7006 文件夹里面,修改配置文件.

启动节点:
/usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7006/redis.conf

执行以下命令,将这个新节点添加到集群里面：
/usr/local/redis-3.0.0/bin/redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000
命令中的 add-node 表示我们要让 redis-trib 将一个节点添加到集群里面.
add-node 之后跟着的是新节点的 IP 地址和端口号.再之后跟着的是集群中任意一个已存在节点的IP地址和端口号,这里我们使用的是 127.0.0.1:7000.
通过 cluster nodes 命令,我们可以确认新节点 127.0.0.1:7006 已经被添加到集群里面了:
redis 127.0.0.1:7006> cluster nodes
f83b31e95b82a2d8c137395a3cee0e2281302470 127.0.0.1:7002 master - 0 1403696051150 5 connected 0-498 5461-5961 10923-16383
af57c7bde78e55ea1223b80ebc7da7493dc36a6e 127.0.0.1:7004 slave 24e4608746554894d3a0723a2c11c9aa1f193212 0 1403696050147 2 connected
c335ad913d635c0eaa88bdd0970e35b323053e2f :0 myself,master - 0 0 0 connected
af4207eb98550ab58c4bf687ac15e5bc89531c47 127.0.0.1:7003 slave 7eb82a407a1326687dbee79848bd9ef86b544689 0 1403696050147 1 connected
24e4608746554894d3a0723a2c11c9aa1f193212 127.0.0.1:7001 master - 0 1403696051150 2 connected 5962-10922
7eb82a407a1326687dbee79848bd9ef86b544689 127.0.0.1:7000 master - 0 1403696050649 1 connected 499-5460
2d0beead6005fd573010a969f4b2fc10fe9c864e 127.0.0.1:7005 slave f83b31e95b82a2d8c137395a3cee0e2281302470 0 1403696049145 5 connected

新节点现在已经连接上了集群,成为集群的一份子,并且可以对客户端的命令请求进行转向了,但是和其他主节点相比,新节点还有两点区别：
新节点没有包含任何数据,因为它没有包含任何哈希桶.
尽管新节点没有包含任何哈希桶,但它仍然是一个主节点,所以在集群需要将某个从节点升级为新的主节点时,这个新节点不会被选中。
接下来,只要使用 redis-trib 程序,将集群中的某些哈希桶移动到新节点里面,新节点就会成为真正的主节点了。
因为使用 redis-trib 移动哈希桶的方法:(详细见reshard)
/usr/local/redis-3.0.0/bin/redis-trib.rb reshard 127.0.0.1:7000
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7003: OK
......
M: c335ad913d635c0eaa88bdd0970e35b323053e2f 127.0.0.1:7006
slots: (0 slots) master
0 additional replica(s)
......
[OK] All nodes agree about slots configuration.

Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 100
What is the receiving node ID? c335ad913d635c0eaa88bdd0970e35b323053e2f
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all
Ready to move 100 slots.
Source nodes:
M: 7eb82a407a1326687dbee79848bd9ef86b544689 127.0.0.1:7000
slots:499-5460 (4962 slots) master
1 additional replica(s)
M: 24e4608746554894d3a0723a2c11c9aa1f193212 127.0.0.1:7001
slots:5962-10922 (4961 slots) master
1 additional replica(s)
M: f83b31e95b82a2d8c137395a3cee0e2281302470 127.0.0.1:7002
slots:0-498,5461-5961,10923-16383 (6461 slots) master
1 additional replica(s)
Destination node:
M: c335ad913d635c0eaa88bdd0970e35b323053e2f 127.0.0.1:7006
slots: (0 slots) master
0 additional replica(s)
Resharding plan:
Moving slot 0 from f83b31e95b82a2d8c137395a3cee0e2281302470
......
Moving slot 5991 from 127.0.0.1:7001 to 127.0.0.1:7006:
[root@lg-2-2-1-n23-84 7006]#
检查:
/usr/local/redis-3.0.0/bin/redis-trib.rb check 127.0.0.1:7000
Connecting to node 127.0.0.1:7000: OK
......
M: c335ad913d635c0eaa88bdd0970e35b323053e2f 127.0.0.1:7006
slots:0-39,499-528,5962-5991 (100 slots) master
0 additional replica(s)
......
[OK] All 16384 slots covered.

现在,让我们来看看,将一个新节点转变为某个主节点的复制品(也即是从节点)的方法.
举个例子,如果我们打算让新节点7007成为127.0.0.1:7006 的从节点.
配置7007的相关配置文件...redis.conf
启动进程: /usr/local/redis-3.0.0/bin/redis-server /usr/local/redis-3.0.0/cluster-test/7007/redis.conf
将新节点加入集群: /usr/local/redis-3.0.0/bin/redis-trib.rb add-node 127.0.0.1:7007 127.0.0.1:7000
......
[OK] All nodes agree about slots configuration.

Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.
Connecting to node 127.0.0.1:7007: OK
Send CLUSTER MEET to node 127.0.0.1:7007 to make it join the cluster.
[OK] New node added correctly.
/usr/local/redis-3.0.0/bin/redis-cli -p 7007
127.0.0.1:7007>cluster replicate c335ad913d635c0eaa88bdd0970e35b323053e2f#这里是7006的ID号,7007作为7006的从.
OK

检查生成的新从节点后,集群状态:
/usr/local/redis-3.0.0/bin/redis-trib.rb check 127.0.0.1:7000
......
Connecting to node 127.0.0.1:7007: OK
......
S: f6ef44729062bd7af3123c026177db9113e66918 127.0.0.1:7007
slots: (0 slots) slave
replicates c335ad913d635c0eaa88bdd0970e35b323053e2f
M: c335ad913d635c0eaa88bdd0970e35b323053e2f 127.0.0.1:7006
slots:0-39,499-528,5962-5991 (100 slots) master
1 additional replica(s)
......
[OK] All nodes agree about slots configuration.

Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.

9.移除节点
略

10.对集群进行重新分片reshard
使用redis-trib.rb执行reshard操作
/usr/local/redis-3.0.0/bin/redis-trib.rb reshard 127.0.0.1:7000 #任意选择一个节点进入集群内操作
......
How many slots do you want to move (from 1 to 16384)? 1000 #计划移动多少个槽位
What is the receiving node ID? f83b31e95b82a2d8c137395a3cee0e2281302470 #要移动槽位的ID号(接收数据的那个)
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all #被移动槽位的源来自哪里
Ready to move 1000 slots.
Source nodes:
M: 7eb82a407a1326687dbee79848bd9ef86b544689 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 24e4608746554894d3a0723a2c11c9aa1f193212 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
Destination node:
M: f83b31e95b82a2d8c137395a3cee0e2281302470 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 5461 from 24e4608746554894d3a0723a2c11c9aa1f193212
Moving slot 5462 from 24e4608746554894d3a0723a2c11c9aa1f193212
......
Moving slot 5961 from 24e4608746554894d3a0723a2c11c9aa1f193212
Moving slot 0 from 7eb82a407a1326687dbee79848bd9ef86b544689
Moving slot 1 from 7eb82a407a1326687dbee79848bd9ef86b544689
......
Moving slot 498 from 7eb82a407a1326687dbee79848bd9ef86b544689
Do you want to proceed with the proposed reshard plan (yes/no)? yes #等待确认是否按照以上推荐方式移动
Moving slot 5461 from 127.0.0.1:7001 to 127.0.0.1:7002:
......
Moving slot 498 from 127.0.0.1:7000 to 127.0.0.1:7002:

移动完毕后,检查集群状态:
[OK] All 16384 slots covered.

11.集群的数据备份与恢复