Akemi

Redis-Cluster扩容与缩容

2024/09/24

Redis-Cluster扩容

1
2
3
4
5
6
7
8
9
10
11
12
环境:
192.168.10.116 redis1
192.168.10.117 redis2
192.168.10.118 redis3

扩容节点：
192.168.10.116 redis
端口也使用8001、8002

3个slots分配到4个slots中
也就是原本每个5460，现在每个4096
每一台分1364过去

slot分配流程

1.目标节点准备导入slots
2.源节点导出slots
3.获取slot下的键
4.迁移键数据
5.循环迁移
6.通知slot分配给目标节点

扩容节点部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
yum -y groupinstall "Development Tools"
yum -y install tcl
wget http://download.redis.io/releases/redis-5.0.5.tar.gz
tar xf redis-5.0.5.tar.gz
cd redis-5.0.5/
make
make install

mkdir -p /usr/local/redis-cluster/8001
mkdir -p /usr/local/redis-cluster/8002
cat >redis-cluster.conf<<EOF
port 8001
daemonize yes
pidfile "/var/run/redis_8001.pid"
dir /usr/local/redis-cluster/8001
cluster-enabled yes #设置以集群模式启动
cluster-config-file nodes-8001.conf #集群节点信息文件
cluster-node-timeout 5000  #离线超时时间
bind 0.0.0.0
protected-mode no
appendonly yes #使用aof
EOF

#写8002从节点的配置文件redis-slave.conf
cat >redis-slave.conf<<EOF
port 8002
daemonize yes
pidfile "/var/run/redis_8002.pid"
dir /usr/local/redis-cluster/8002
cluster-enabled yes
cluster-config-file nodes-8002.conf
cluster-node-timeout 5000
bind 0.0.0.0
protected-mode no 
appendonly yes
EOF

redis-server redis-cluster.conf
redis-server redis-slave.conf

扩容redis-cluster实例

给集群写点数据，用以测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cat >/usr/local/redis-cluster/redis.sh<<'EOF'
#!/bin/bash    
# Redis 端口  
PORT=8001 
# 遍历 1 到 1002  
for (( i=1; i<=1002; i++ ))  
do  
    # 构造键和值  
    KEY="k_$i"  
    VALUE="v_$i"  
    # 使用 redis-cli 设置键值对,-c表示集群模式
    redis-cli -c -p $PORT set "$KEY" "$VALUE"  
done  
EOF
chmod +x /usr/local/redis-cluster/redis.sh
/usr/local/redis-cluster/redis.sh

扩容主节点、从节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#检查cluster状态
redis-cli --cluster check 192.168.10.116:8001
如有报错就做不了

#加入主节点
redis-cli --cluster add-node 新主节点:端口 任意旧节点:端口
redis-cli --cluster add-node 192.168.10.102:8001 192.168.10.116:8001

#为主节点加入从节点
redis-cli --cluster add-node 新从节点:端口 主节点:端口 --cluster-slave --cluster-master-id 主节点id
redis-cli --cluster add-node 192.168.10.102:8002 192.168.10.102:8001 --cluster-slave --cluster-master-id 489e3ca8187b1a51f46418fa8d0a36092c8b2051

redis-cli -p 8001 CLUSTER nodes | grep 192.168.10.102
489e3ca8187b1a51f46418fa8d0a36092c8b2051 192.168.10.102:8001@18001 master - 0 1727099368000 0 connected
e04ae432959e1a520461b1ae98c468285f87db7f 192.168.10.102:8002@18002 slave 489e3ca8187b1a51f46418fa8d0a36092c8b2051 0 1727099368593 0 connected

重新分配slot hash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
redis-cli --cluster reshard 任意节点
↓
redis-cli --cluster reshard 192.168.10.116:8001

交互式输入：
How many slots do you want to move (from 1 to 16384)?————分多少个slot 
4096

What is the receiving node ID?————接收节点的id、即新主节点id
489e3ca8187b1a51f46418fa8d0a36092c8b2051

Source node #1:————slot来源
all
或者
写三个节点的id、回车

Do you want to proceed with the proposed reshard plan (yes/no)?
yes

集群状态验证

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
check查看是否有错误
redis-cli --cluster check 192.168.10.116:8001 | grep OK
[OK] 1002 keys in 4 masters.
[OK] All nodes agree about slots configuration.
[OK] All 16384 slots covered.

可以看到四个节点都有4096个slot了
redis-cli --cluster check 192.168.10.116:8001 | grep 4096
192.168.10.116:8001 (c37cd7a3...) -> 250 keys | 4096 slots | 1 slaves.
192.168.10.102:8001 (489e3ca8...) -> 256 keys | 4096 slots | 1 slaves.
192.168.10.118:8001 (68c3eb97...) -> 250 keys | 4096 slots | 1 slaves.
192.168.10.117:8001 (68d676f1...) -> 246 keys | 4096 slots | 1 slaves.
   slots:[1365-5460] (4096 slots) master
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
   slots:[12288-16383] (4096 slots) master
   slots:[6827-10922] (4096 slots) master
 
验证数据
redis-cli -h 192.168.10.102 -p 8001
KEYS *

Redis-Cluster收缩

反向的扩容,192.168.10.102成为slot的源节点

1.迁移槽
2.删除节点

第一个节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
4096/3≈1365.33
多出来的节点需要向其他节点分1365个slot

redis-cli --cluster reshard 192.168.10.116:8001

交互式：
How many slots do you want to move (from 1 to 16384)?第1,2次1365，第3次1366
1365

What is the receiving node ID?谁接收，下面这个id是116的node id。第2、3节点要输另外的
c37cd7a35beb35c7cf648d54c88f58514fd8dbdd

Source node  谁发送，输102的node id
489e3ca8187b1a51f46418fa8d0a36092c8b2051
done

第2、3节点

从执行之后的表中就能看出哪个节点的slots多，哪个少

按照1节点迁移时标注的注释，进行交互式填入即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
redis-cli --cluster reshard 192.168.10.116:8001
>>> Performing Cluster Check (using node 192.168.10.116:8001)
M: c37cd7a35beb35c7cf648d54c88f58514fd8dbdd 192.168.10.116:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: c35ba3d5788e974ad6830f5dbdeaa5fa0303bd2e 192.168.10.118:8002
   slots: (0 slots) slave
   replicates 68d676f1a14073941cfeb46e88f281bc85f61c1d
S: 8091088d4c52b74d29185729ff820f1e821bdb30 192.168.10.117:8002
   slots: (0 slots) slave
   replicates c37cd7a35beb35c7cf648d54c88f58514fd8dbdd
M: 489e3ca8187b1a51f46418fa8d0a36092c8b2051 192.168.10.102:8001
   slots:[6826],[10923-12287] (1366 slots) master
   1 additional replica(s)
M: 68c3eb97a6a9d531f83690af8f780f4ed2af73c4 192.168.10.118:8001
   slots:[5461-6825],[12288-16383] (5461 slots) master
   1 additional replica(s)
S: e04ae432959e1a520461b1ae98c468285f87db7f 192.168.10.102:8002
   slots: (0 slots) slave
   replicates 489e3ca8187b1a51f46418fa8d0a36092c8b2051
S: 8324825e7ecf81cf4ae4bf2604b022b8a8f2e068 192.168.10.116:8002
   slots: (0 slots) slave
   replicates 68c3eb97a6a9d531f83690af8f780f4ed2af73c4
M: 68d676f1a14073941cfeb46e88f281bc85f61c1d 192.168.10.117:8001
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

迁移后状态查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
redis-cli --cluster check 192.168.10.116 8001
192.168.10.116:8001 (c37cd7a3...) -> 340 keys | 5461 slots | 1 slaves.
192.168.10.102:8001 (489e3ca8...) -> 0 keys | 0 slots | 0 slaves.
192.168.10.118:8001 (68c3eb97...) -> 331 keys | 5461 slots | 1 slaves.
192.168.10.117:8001 (68d676f1...) -> 331 keys | 5462 slots | 2 slaves.
[OK] 1002 keys in 4 masters.
0.06 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.10.116:8001)
M: c37cd7a35beb35c7cf648d54c88f58514fd8dbdd 192.168.10.116:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: c35ba3d5788e974ad6830f5dbdeaa5fa0303bd2e 192.168.10.118:8002
   slots: (0 slots) slave
   replicates 68d676f1a14073941cfeb46e88f281bc85f61c1d
S: 8091088d4c52b74d29185729ff820f1e821bdb30 192.168.10.117:8002
   slots: (0 slots) slave
   replicates c37cd7a35beb35c7cf648d54c88f58514fd8dbdd
M: 489e3ca8187b1a51f46418fa8d0a36092c8b2051 192.168.10.102:8001
   slots: (0 slots) master
M: 68c3eb97a6a9d531f83690af8f780f4ed2af73c4 192.168.10.118:8001
   slots:[5461-6825],[12288-16383] (5461 slots) master
   1 additional replica(s)
S: e04ae432959e1a520461b1ae98c468285f87db7f 192.168.10.102:8002
   slots: (0 slots) slave
   replicates 68d676f1a14073941cfeb46e88f281bc85f61c1d
S: 8324825e7ecf81cf4ae4bf2604b022b8a8f2e068 192.168.10.116:8002
   slots: (0 slots) slave
   replicates 68c3eb97a6a9d531f83690af8f780f4ed2af73c4
M: 68d676f1a14073941cfeb46e88f281bc85f61c1d 192.168.10.117:8001
   slots:[6826-12287] (5462 slots) master
   2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

节点下线

删除节点

如上可知，下面这两个节点需要删除
M: 489e3ca8187b1a51f46418fa8d0a36092c8b2051 192.168.10.102:8001
slots: (0 slots) master
S: e04ae432959e1a520461b1ae98c468285f87db7f 192.168.10.102:8002
slots: (0 slots) slave

1
2
3
4
redis-cli --cluster del-node 192.168.10.102:8001 \
489e3ca8187b1a51f46418fa8d0a36092c8b2051
redis-cli --cluster del-node 192.168.10.102:8002 \
e04ae432959e1a520461b1ae98c468285f87db7f

状态查看

状态正常

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
redis-cli --cluster check 192.168.10.116:8001
192.168.10.116:8001 (c37cd7a3...) -> 340 keys | 5461 slots | 1 slaves.
192.168.10.118:8001 (68c3eb97...) -> 331 keys | 5461 slots | 1 slaves.
192.168.10.117:8001 (68d676f1...) -> 331 keys | 5462 slots | 1 slaves.
[OK] 1002 keys in 3 masters.
0.06 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.10.116:8001)
M: c37cd7a35beb35c7cf648d54c88f58514fd8dbdd 192.168.10.116:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: c35ba3d5788e974ad6830f5dbdeaa5fa0303bd2e 192.168.10.118:8002
   slots: (0 slots) slave
   replicates 68d676f1a14073941cfeb46e88f281bc85f61c1d
S: 8091088d4c52b74d29185729ff820f1e821bdb30 192.168.10.117:8002
   slots: (0 slots) slave
   replicates c37cd7a35beb35c7cf648d54c88f58514fd8dbdd
M: 68c3eb97a6a9d531f83690af8f780f4ed2af73c4 192.168.10.118:8001
   slots:[5461-6825],[12288-16383] (5461 slots) master
   1 additional replica(s)
S: 8324825e7ecf81cf4ae4bf2604b022b8a8f2e068 192.168.10.116:8002
   slots: (0 slots) slave
   replicates 68c3eb97a6a9d531f83690af8f780f4ed2af73c4
M: 68d676f1a14073941cfeb46e88f281bc85f61c1d 192.168.10.117:8001
   slots:[6826-12287] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

原文作者：王盛

原文链接：https://akemi.zj.cn/2024/09/24/Redis-Scaling/

发表日期：September 24th 2024, 12:55:53 am

更新日期：February 20th 2025, 6:37:27 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. Redis-Cluster扩容
    1. 1.1. slot分配流程
    2. 1.2. 扩容节点部署
    3. 1.3. 扩容redis-cluster实例
  2. 2. Redis-Cluster收缩
    1. 2.1. 第一个节点
    2. 2.2. 第2、3节点
    3. 2.3. 节点下线
Total : 203
2025
2024
2023
Linux 云原生 云计算 开发 mysql