广州网匠营销型网站建设公司,吉林智能网站建设找哪家,重庆网站建设快忻,lol关键词查询k8s部署redis6节点#xff0c;组成3主3从集群模式
一般来说#xff0c;redis部署有三种模式。 单实例模式#xff0c;一般用于测试环境。 哨兵模式 集群模式后两者用于生产部署
哨兵模式 在redis3.0以前#xff0c;要实现集群一般是借助哨兵sentinel工具来监控master节点…k8s部署redis6节点组成3主3从集群模式
一般来说redis部署有三种模式。 单实例模式一般用于测试环境。 哨兵模式 集群模式后两者用于生产部署
哨兵模式 在redis3.0以前要实现集群一般是借助哨兵sentinel工具来监控master节点的状态。 如果master节点异常则会做主从切换将某一台slave作为master。 引入了哨兵节点部署更复杂维护成本也比较高并且性能和高可用性等各方面表现一般。
集群模式 3.0 后推出的 Redis 分布式集群解决方案 主节点提供读写操作从节点作为备用节点不提供请求只作为故障转移使用 如果master节点异常也是会自动做主从切换将slave切换为master。
总的来说集群模式明显优于哨兵模式 一、redis集群搭建
RC、Deployment、DaemonSet都是面向无状态的服务它们所管理的Pod的IP、名字启停顺序等都是随机的而StatefulSet是什么顾名思义有状态的集合管理所有有状态的服务比如MySQL、MongoDB集群等。
StatefulSet本质上是Deployment的一种变体在v1.9版本中已成为GA版本它为了解决有状态服务的问题它所管理的Pod拥有固定的Pod名称启停顺序在StatefulSet中Pod名字称为网络标识(hostname)还必须要用到共享存储。
在Deployment中与之对应的服务是service而在StatefulSet中与之对应的headless serviceheadless service即无头服务与service的区别就是它没有Cluster IP解析它的名称时将返回该Headless Service对应的全部Pod的Endpoint列表。
除此之外StatefulSet在Headless Service的基础上又为StatefulSet控制的每个Pod副本创建了一个DNS域名这个域名的格式为 ( p o d . n a m e ) . (pod.name). (pod.name).(headless server.name).${namespace}.svc.cluster.local 也即是说对于有状态服务我们最好使用固定的网络标识如域名信息来标记节点当然这也需要应用程序的支持如Zookeeper就支持在配置文件中写入主机域名。
StatefulSet基于Headless Service即没有Cluster IP的Service为Pod实现了稳定的网络标志包括Pod的hostname和DNS Records在Pod重新调度后也保持不变。同时结合PV/PVCStatefulSet可以实现稳定的持久化存储就算Pod重新调度后还是能访问到原先的持久化数据。
以下为使用StatefulSet部署Redis的架构无论是Master还是Slave都作为StatefulSet的一个副本并且数据通过PV进行持久化对外暴露为一个Service接受客户端请求。 1.redis.conf配置文件参考
说明我们知道redis默认目录是/var/lib/redis/和/etc/redis/同时官方在构建redis镜像时默认工作目录在/data目录所以本篇为了规范redis数据存放目录将redis.conf挂载到/etc/redis/下其他redis日志文件、数据文件全部放到/data目录下。
#[rootmaster redis]# vi redis.conf #编写一个redis.conf配置文件
#[rootmaster redis]# grep -Ev $^|# redis.conf #下面是redis.conf配置文件
bind 0.0.0.0
protected-mode yes
port 6379 #redis端口,为了安全设置为6379端口
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no #redis是否以后台模式运行,必须设置no
supervised no
pidfile /data/redis.pid #redis的pid文件,放到/data目录下
loglevel notice
logfile /data/redis_log #redis日志文件,放到/data目录下
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb #这个文件会放在dir定义的/data目录
dir /data #数据目录
masterauth iloveyou #redis集群各节点相互认证的密码,必须配置和下面的requirepass一致
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
requirepass iloveyou #redis的密码
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly no
appendfilename appendonly.aof #这个文件会放在dir定义的/data目录
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
cluster-enabled yes #是否启用集群模式必须去掉注释设为yes
cluster-config-file nodes.conf #这个文件会放在dir定义的/data目录
cluster-node-timeout 15000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes 2.创建statefulsets有状态应用
redis集群一般可以使用deployment和statefulsets这里使用statefulsets有状态应用来创建redis创建sts有状态应用需要有一个headless service同时在sts中挂载configmap卷使用动态分配pv用于redis数据持久化。
cat 03-redis-cluster-sts.yaml
cat 03-redis-cluster-sts.yaml eof
apiVersion: v1
kind: ConfigMap
metadata:name: redis-cluster-config
data:redis-cluster.conf: |daemonize nosupervised noprotected-mode nobind 0.0.0.0port 6379cluster-announce-bus-port 16379cluster-enabled yesappendonly yescluster-node-timeout 5000dir /datacluster-config-file /data/nodes.confrequirepass iloveyoumasterauth iloveyoupidfile /data/redis.pid loglevel noticelogfile /data/redis_log
---
apiVersion: v1
kind: Service
metadata:name: redis-cluster-service
spec:selector:app: redis-clusterclusterIP: Noneports:- name: redis-6379port: 6379- name: redis-16379port: 16379
---
apiVersion: v1
kind: Service
metadata:name: redis-cluster-service-access
spec:selector:app: redis-clustertype: NodePortsessionAffinity: Noneports:- name: redis-6379port: 6379targetPort: 6379nodePort: 30202
---
apiVersion: apps/v1
kind: StatefulSet
metadata:labels:app: redis-clustername: redis-cluster
spec:serviceName: redis-cluster-servicereplicas: 6selector:matchLabels:app: redis-clustertemplate:metadata:labels:app: redis-clusterspec:terminationGracePeriodSeconds: 30containers:- name: redisimage: redis:6.2.6imagePullPolicy: IfNotPresentterminationMessagePath: /dev/termination-logterminationMessagePolicy: Filecommand: [ redis-server, /etc/redis/redis-cluster.conf ]args:- --cluster-announce-ip- $(POD_IP)env:- name: HOST_IPvalueFrom:fieldRef:fieldPath: status.hostIP- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: TZvalue: Asia/Shanghaiports:- name: rediscontainerPort: 6379protocol: TCP- name: clustercontainerPort: 16379protocol: TCPvolumeMounts:- name: redis-confmountPath: /etc/redis- name: pvc-datamountPath: /datavolumes:- name: timezonehostPath:path: /usr/share/zoneinfo/Asia/Shanghai- name: redis-confconfigMap:name: redis-cluster-configitems:- key: redis-cluster.confpath: redis-cluster.confvolumeClaimTemplates:- metadata:name: pvc-dataspec:accessModes: [ ReadWriteOnce ]resources:requests:storage: 200MaccessModes:- ReadWriteOnce #访问模式为RWOstorageClassName: nfs-boge #指定使用的存储类实现动态分配pv
eofkubectl apply -f 03-redis-cluster-sts.yaml二、构建3主3从集群模式
6个pod已经创建完毕状态都是running下面将6个pod 组成redis集群3主3从模式。
命令说明
–replicas 1或者–cluster-replicas 1 指定集群中每个master的副本个数为1此时节点总数 ÷ (replicas 1) 得到的就是master的数量。因此节点列表中的前n个就是master其它节点都是slave节点随机分配到不同master
#在redis任意一个pod执行初始化命令,可以进入到pod里面执行也可以直接在外面执行
#其中为了获取每个pod的ip,使用
# kubectl get pods -l appredis-cluster -o jsonpath{range.items[*]}{.status.podIP}:6379 {end}
#本次采用自动创建redis的形式,也就是说不指定哪个是主哪个是从节点,让redis自动分配,生产环境中也建议使用该种方式[rootmaster redis]# kubectl exec -it redis-cluster-0 -- redis-cli -a iloveyou --cluster create --cluster-replicas 1 $(kubectl get pods -l appredis-cluster -o jsonpath{range.items[*]}{.status.podIP}:6379 {end})
Warning: Using a password with -a or -u option on the command line interface may not be safe.Performing hash slots allocation on 6 nodes...
Master[0] - Slots 0 - 5460
Master[1] - Slots 5461 - 10922
Master[2] - Slots 10923 - 16383
Adding replica 10.244.2.22:6379 to 10.244.2.20:6379
Adding replica 10.244.1.20:6379 to 10.244.1.18:6379
Adding replica 10.244.2.21:6379 to 10.244.1.19:6379
M: 972b376e8cc658b8bf5f2a1a3294cbe2c84ee852 10.244.2.20:6379slots:[0-5460] (5461 slots) master
M: ac6cc9dd3a86cf370333d36933c99df5f13f42ab 10.244.1.18:6379slots:[5461-10922] (5462 slots) master
M: 18b4ceacd3222e546ab59e041e4ae50e736c5c26 10.244.1.19:6379slots:[10923-16383] (5461 slots) master
S: 8394ceff0b32fc7119b65704ea78e9b5bbc2fbd7 10.244.2.21:6379replicates 18b4ceacd3222e546ab59e041e4ae50e736c5c26
S: 565f9f9931323f8ac0376b7a7ec701f0a2955e8b 10.244.2.22:6379replicates 972b376e8cc658b8bf5f2a1a3294cbe2c84ee852
S: 5c1270743b6a5f81003da4402f39c360631a2d0f 10.244.1.20:6379replicates ac6cc9dd3a86cf370333d36933c99df5f13f42ab
Can I set the above configuration? (type yes to accept): yes #输入yes表示接受redis给我们自动分配的槽点,主从也是redis任意指定的Nodes configuration updatedAssign a different config epoch to each nodeSending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..Performing Cluster Check (using node 10.244.2.20:6379)
M: 972b376e8cc658b8bf5f2a1a3294cbe2c84ee852 10.244.2.20:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
S: 5c1270743b6a5f81003da4402f39c360631a2d0f 10.244.1.20:6379slots: (0 slots) slavereplicates ac6cc9dd3a86cf370333d36933c99df5f13f42ab
M: 18b4ceacd3222e546ab59e041e4ae50e736c5c26 10.244.1.19:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
M: ac6cc9dd3a86cf370333d36933c99df5f13f42ab 10.244.1.18:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: 565f9f9931323f8ac0376b7a7ec701f0a2955e8b 10.244.2.22:6379slots: (0 slots) slavereplicates 972b376e8cc658b8bf5f2a1a3294cbe2c84ee852
S: 8394ceff0b32fc7119b65704ea78e9b5bbc2fbd7 10.244.2.21:6379slots: (0 slots) slavereplicates 18b4ceacd3222e546ab59e041e4ae50e736c5c26
[OK] All nodes agree about slots configuration.Check for open slots...Check slots coverage...
[OK] All 16384 slots covered.
[rootmaster redis]# 1.验证redis
#往集群里面写入一百条数据。
for line in {1..100};do kubectl exec -it redis-cluster-0 -- redis-cli -c -p 6379 -a iloveyou -c set ops_${line} ${line}; done#读取数据看一下是否能读取到。
for line in {1..100};do kubectl exec -it redis-cluster-1 -- redis-cli -c -p 6379 -a iloveyou -c get ops_${line}; done#登入redis
kubectl exec -it redis-cluster-0 -- redis-cli -c -p 6379 -a iloveyou#查看redis信息
127.0.0.1:6379 cluster info#检查redis集群信息
127.0.0.1:6379 cluster nodes#插入数据
127.0.0.1:6379 set num 111#遍历所有的key
127.0.0.1:6379 keys *#删除key
127.0.0.1:6379 del num
#登入redis0
kubectl exec -it redis-cluster-0 -- redis-cli -c -p 6379 -a iloveyou#遍历所有的key
127.0.0.1:6379 keys *#登入redis1
kubectl exec -it redis-cluster-1 -- redis-cli -c -p 6379 -a iloveyou#遍历所有的key
127.0.0.1:6379 keys *#登入redis2
kubectl exec -it redis-cluster-2 -- redis-cli -c -p 6379 -a iloveyou#遍历所有的key
127.0.0.1:6379 keys *2.删除pod验证 2.1 验证集群信息
[rootk8s-master01 redis]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-cluster-0 1/1 Running 0 61m 100.125.152.57 k8s-node02 none none
redis-cluster-1 1/1 Running 0 33m 100.97.125.39 k8s-node01 none none
redis-cluster-2 1/1 Running 0 60m 100.125.152.22 k8s-node02 none none
redis-cluster-3 1/1 Running 0 60m 100.97.125.10 k8s-node01 none none
redis-cluster-4 1/1 Running 0 60m 100.125.152.41 k8s-node02 none none
redis-cluster-5 1/1 Running 0 60m 100.97.125.57 k8s-node01 none none
[rootk8s-master01 redis]#
#登入rediskubectl exec -it redis-cluster-0 – redis-cli -c -p 6379 -a iloveyou
#检查redis集群信息
127.0.0.1:6379 cluster nodes
e4189ba4094320d4f379fb0cbc78cac6cdb3abe6 100.97.125.10:637916379 slave 0e3cae56f05f211c35b4401f6a0e61454ca4f64a 0 1693891070028 3 connected
9cdff4df51646dd440e5005bac7f0cb5f4470d9b 100.125.152.41:637916379 slave 0a228f6fbd7fc75247180ac31516146e8abd0466 0 1693891069927 1 connected
93ccc4c795f74da6acf84821bf372a2414375f55 100.97.125.39:637916379 master - 0 1693891070430 2 connected 5461-10922
ff960d1a1aca448a391b69da155947b4c627677e 100.97.125.57:637916379 slave 93ccc4c795f74da6acf84821bf372a2414375f55 0 1693891069000 2 connected
0e3cae56f05f211c35b4401f6a0e61454ca4f64a 100.125.152.22:637916379 master - 0 1693891069426 3 connected 10923-16383
0a228f6fbd7fc75247180ac31516146e8abd0466 100.125.152.57:637916379 myself,master - 0 1693891069000 1 connected 0-5460
2.2.删除pod
[rootk8s-master01 redis]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-cluster-0 1/1 Running 0 61m 100.125.152.57 k8s-node02 none none
redis-cluster-1 1/1 Running 0 33m 100.97.125.39 k8s-node01 none none
redis-cluster-2 1/1 Running 0 60m 100.125.152.22 k8s-node02 none none
redis-cluster-3 1/1 Running 0 60m 100.97.125.10 k8s-node01 none none
redis-cluster-4 1/1 Running 0 60m 100.125.152.41 k8s-node02 none none
redis-cluster-5 1/1 Running 0 60m 100.97.125.57 k8s-node01 none none
[rootk8s-master01 redis]# kubectl delete po redis-cluster-5 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod redis-cluster-5 force deleted
[rootk8s-master01 redis]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-cluster-0 1/1 Running 0 62m 100.125.152.57 k8s-node02 none none
redis-cluster-1 1/1 Running 0 35m 100.97.125.39 k8s-node01 none none
redis-cluster-2 1/1 Running 0 62m 100.125.152.22 k8s-node02 none none
redis-cluster-3 1/1 Running 0 62m 100.97.125.10 k8s-node01 none none
redis-cluster-4 1/1 Running 0 61m 100.125.152.41 k8s-node02 none none
redis-cluster-5 1/1 Running 0 6s 100.97.125.37 k8s-node01 none none
[rootk8s-master01 redis]# 2.2 验证redis集群
[rootk8s-master01 redis]# kubectl exec -it redis-cluster-1 -- redis-cli -c -p 6379 -a iloveyou
Warning: Using a password with -a or -u option on the command line interface may not be safe.
127.0.0.1:6379 cluster nodes
e4189ba4094320d4f379fb0cbc78cac6cdb3abe6 100.97.125.10:637916379 slave 0e3cae56f05f211c35b4401f6a0e61454ca4f64a 0 1693891297573 3 connected
9cdff4df51646dd440e5005bac7f0cb5f4470d9b 100.125.152.41:637916379 slave 0a228f6fbd7fc75247180ac31516146e8abd0466 0 1693891297073 1 connected
93ccc4c795f74da6acf84821bf372a2414375f55 100.97.125.39:637916379 myself,master - 0 1693891297000 2 connected 5461-10922
0e3cae56f05f211c35b4401f6a0e61454ca4f64a 100.125.152.22:637916379 master - 0 1693891297573 3 connected 10923-16383
ff960d1a1aca448a391b69da155947b4c627677e 100.97.125.37:637916379 slave 93ccc4c795f74da6acf84821bf372a2414375f55 0 1693891297073 2 connected
0a228f6fbd7fc75247180ac31516146e8abd0466 100.125.152.57:637916379 master - 0 1693891296571 1 connected 0-5460
127.0.0.1:6379
查询插入数据验证
del num#读取数据看一下是否能读取到。
for line in {1..100};do kubectl exec -it redis-cluster-4 -- redis-cli -c -p 6379 -a iloveyou -c get ops_${line}; done#往集群里面写入一百条数据。
for line in {1..200};do kubectl exec -it redis-cluster-5 -- redis-cli -c -p 6379 -a iloveyou -c set ops_${line} ${line}; donefor line in {1..200};do kubectl exec -it redis-cluster-2 -- redis-cli -c -p 6379 -a iloveyou -c del ops_${line}; done#往集群里面写入32条数据。
for line in {1..32};do kubectl exec -it redis-cluster-5 -- redis-cli -c -p 6379 -a iloveyou -c set ops_${line} ${line}; done
