1200字范文,内容丰富有趣,写作的好帮手!
1200字范文 > nginx和keeplive实现负载均衡高可用

nginx和keeplive实现负载均衡高可用

时间:2021-05-24 23:36:37

相关推荐

nginx和keeplive实现负载均衡高可用

一、 Keeplive服务介绍

Keeplive期初是专门为LVS设计的,专门用来监控LVS集群系统中各个服务节点的状态,后来又加入VRRP的功能,因此除了配合LVS服务以外,也可以作为其他服务(nginx,haroxy)的高可用软件,VRRP是Virtual Router Redundancy Protocol(虚拟路由冗余协议)的缩写,VRRP出现的目的就是为了解决静态路由出现的单点故障问题,它能够保证网络的不间断,稳定的运行。所以keepalive一方面具有LVS cluster nodes healthchecks功能,另一方面也具有LVS directors failover

1.1 Keepalived的用途

Keepalive服务的两大用途:healthcheck和failover

ha failover功能:实现LB Master主机和Backup主机之间故障转移和自动切换

这是针对有两个负载均衡器Direator同时工作而采取的故障转移措施,当主负载均衡器失效或者出现故障时,备份的负载均衡器(BACKUP)将自动接管主负载均衡器的所有工作(vip资源以及相关服务):一旦主负载均衡器故障恢复,MASTER又会接管回它原来的工作,二备份复杂均衡器(BACKUP)会释放master是小事它接管的工作,此时两者将恢复到最初各自的角色

1.2 LVS cluster nodes healthchecks功能

在keeplive.conf配置记忆可以实现LVS的功能

keeplive可以对LVS下面的集群节点进行健康检查

rs healthcheck功能:负载均衡定期检查RS的可用性决定是否给其分发请求

当虚拟的服务器中的某一个甚至是几个真实的服务器同时出现故障无法提供服务时,负载均衡器会自动将失效的RS服务器从转发队列中清除出去,从而保证用户的访问不收影响;当故障的RS服务器被修复后,系统又自动的将他们加入转发队列,分发请求提供正常服务。

工作原理

1.3 keepalive故障切换转换原理

Keepalived高可用对之间是通过 VRRP进行通信的, VRRP是遑过竞选机制来确定主备的,主的优先级高于备,因此,工作时主会优先获得所有的资源,备节点处于等待状态,当主挂了的时候,备节点就会接管主节点的资源,然后顶替主节点对外提供服务。

在 Keepalived服务对之间,只有作为主的服务器会一直发送 VRRP广播包,告诉备它还活着,此时备不会枪占主,当主不可用时,即备监听不到主发送的广播包时,就会启动相关服务接管资源,保证业务的连续性.接管速度最快可以小于1秒。

1.4 VRRP协议的简单介绍

1) VRRP,全称 Virtual Router Redundancy Protocol,中文名为虚拟路由冗余协议,VRRP的出现是为了解决静态路由的单点故障。

2) VRRP是通过一种竟选协议机制来将路由任务交给某台 VRRP路由器的。

3) VRRP用 IP多播的方式(默认多播地址(224.0_0.18))实现高可用对之间通信。

4) 工作时主节点发包,备节点接包,当备节点接收不到主节点发的数据包的时候,就启动接管程序接管主节点的开源。备节点可以有多个,通过优先级竞选,但一般 Keepalived系统运维工作中都是一对。

5) VRRP使用了加密协议加密数据,但Keepalived官方目前还是推荐用明文的方式配置认证类型和密码

二、配置Keepalived实现高可用

2.1 安装keepalive

[root@lb01 ~]# cd /usr/local/src/

[root@lb01 src]# wget /software/keepalived-2.0.15.tar.gz

[root@lb01 src]# tar -xf keepalived-2.0.15.tar.gz

[root@lb01 src]# cd keepalived-2.0.15

[root@lb01 keepalived-2.0.15]# ./configure

Linker flags : -pieExtra Lib: -lm -lcrypto -lssl Use IPVS Framework : YesIPVS use libnl : NoIPVS syncd attributes : NoIPVS 64 bit stats : NoHTTP_GET regex support : Nofwmark socket support : YesUse VRRP Framework : YesUse VRRP VMAC : YesUse VRRP authentication : YesWith ip rules/routes: YesWith track_process : YesWith linkbeat : YesUse BFD Framework : NoSNMP vrrp support : NoSNMP checker support: NoSNMP RFCv2 support : NoSNMP RFCv3 support : NoDBUS support : NoSHA1 support : NoUse JSON output: Nolibnl version : NoneUse IPv4 devconf : NoUse iptables : YesUse libiptc : NoUse libipset : NoUse nftables : Noinit type: systemdStrict config checks: NoBuild genhash : YesBuild documentation: No

[root@lb01 keepalived-2.0.15]# make

[root@lb01 keepalived-2.0.15]# make install

lb02相同操作

[root@lb02 ~]# cd /usr/local/src/[root@lb02 src]# wget /software/keepalived-2.0.15.tar.gz[root@lb02 src]# tar -xf keepalived-2.0.15.tar.gz[root@lb02 src]# cd keepalived-2.0.15[root@lb02 keepalived-2.0.15]# ./configure [root@lb02 keepalived-2.0.15]# make[root@lb02 keepalived-2.0.15]# make install

[root@lb01 keepalived-2.0.15]# vim /usr/lib/systemd/system/keepalived.service

[Unit]

Description=LVS and VRRP High Availability Monitor

After=syslog.target network-online.target

[Service]

Type=forking

PIDFile=/var/run/keepalived.pid

KillMode=process

EnvironmentFile=-/etc/sysconfig/keepalived

ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS

ExecReload=/bin/kill -HUP $MAINPID

[Install]

WantedBy=multi-user.target

2.2 配置文件

[root@lb01 keepalived-2.0.15]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived#!注释global_defs { #全局变量notification_email {283365585@#收件人} #邮件发件人notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 #邮件服务器地址smtp_connect_timeout 30#超时时间router_id LVS_01vrrp_skip_check_adv_addrvrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0}vrrp_instance VI_1 {#keepalive或者vrrp的一个实例state MASTER #状态interface ens33#通信端口virtual_router_id 51 #实例IDpriority 150 #优先级advert_int 1 #心跳的间隔authentication {#服务器之间通过密码验证auth_type PASSauth_pass 1111} virtual_ipaddress {10.0.0.131 #VIP} }

2.3 启动看效果

[root@lb01 keepalived]# systemctl start keepalived

[root@lb01 keepalived]# ps -ef|grep keep

[root@lb01 keepalived]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:12:2e:59 brd ff:ff:ff:ff:ff:ffinet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33valid_lft 1085sec preferred_lft 1085secinet10.0.0.131/24scope global ens33valid_lft forever preferred_lft foreverinet6 fe80::8068:96e2:b57b:be1d/64 scope link valid_lft forever preferred_lft forever3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:12:2e:63 brd ff:ff:ff:ff:ff:ff

2.4 keepalive的VIP实现形式

[root@lb01 keepalived]# ifconfig ens33:0 10.0.0.18 up

[root@lb01 keepalived]# ip addr add 10.0.0.19 dev ens33

[root@lb01 keepalived]# ip addr

inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33valid_lft 1583sec preferred_lft 1583secinet 10.0.0.131/24 scope global ens33valid_lft forever preferred_lft foreverinet 10.0.0.18/8 brd 10.255.255.255 scope global ens33:0valid_lft forever preferred_lft foreverinet 10.0.0.19/32 scope global ens33valid_lft forever preferred_lft foreverinet6 fe80::8068:96e2:b57b:be1d/64 scope link valid_lft forever preferred_lft forever

[root@lb01 keepalived]# ip addr del 10.0.0.19 dev ens33

[root@lb01 keepalived]# ifconfig ens33:0 10.0.0.18 down

[root@lb01 keepalived]# scp /etc/keepalived/keepalived.conf 172.25.254.133:/etc/keepalived/

[root@lb02 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalivedglobal_defs {notification_email {283365585@}notification_email_from Alexandre.Cassen@firewall.locsmtp_server 192.168.200.1smtp_connect_timeout 30router_id LVS_02vrrp_skip_check_adv_addrvrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0}vrrp_instance VI_1 {state BACKUPinterface ens33virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {10.0.0.131/24}}

[root@lb02 ~]# systemctl start keepalived

[root@lb02 ~]# ps -ef |grep keep

配置成功

2.5 检测keepalibve效果

关闭MASTER的keepalive服务

[root@lb01 keepalived]# systemctl stop keepalived

[root@lb01 keepalived]# ip addr|grep 10.0.0.131

查看BACKUP端,是否有10.0.0.131

[root@lb02 ~]# ip addr|grep 10.0.0.131

[root@lb01 keepalived]# systemctl start keepalived

成功

三、 结合nginx实现高可用

3.1 配置

[root@lb01 keepalived]# cd /usr/local/nginx/conf/

[root@lb01 conf]# vim nginx.conf

worker_processes 1;events {worker_connections 1024;}http {include mime.types;default_type application/octet-stream;sendfile on;keepalive_timeout 65;upstream web_pools {server 172.25.254.134:80 weight=5;server 172.25.254.135:80 weight=5;# server 172.25.254.158:80 weight=5 backup;}server {listen 80;server_name ;location / {# root html;# index index.html index.htm;proxy_set_header Host $host;proxy_pass http://web_pools; }}}

[root@lb01 conf]# scp nginx.conf 172.25.254.133:/usr/local/nginx/conf/

[root@lb01 conf]# nginx -s reload[root@lb01 conf]# curl 172.25.254.134172.25.254.134[root@lb01 conf]# curl 172.25.254.135172.25.254.135[root@lb01 conf]# nginx -s reload[root@lb02 ~]# curl 172.25.254.134172.25.254.134[root@lb02 ~]# curl 172.25.254.135172.25.254.135

在获取到VIP后,不在同一网段,为了方便测试,把VIP设置为172.25.254.254

[root@lb01 ~]# ip addr|grep 172.25.254.254

3.2 发现不能访问的问题

访问测试,发现不能访问,也不能ping通

[root@lb01 conf]# curl 172.25.254.254

curl: (7) Failed connect to 172.25.254.254:80; Connection timed out

[root@lb01 conf]# ping 172.25.254.254

2 packets transmitted, 0 received, 100% packet loss, time 999ms

解决

[root@lb01 conf]# vim /etc/keepalived/keepalived.conf

# vrrp_strict # 注释掉vrrp_strict

[root@lb01 conf]# systemctl restart keepalived

3.3 测试

[root@lb01 conf]# curl 172.25.254.254172.25.254.135[root@lb01 conf]# curl 172.25.254.254172.25.254.134[root@lb01 conf]# curl 172.25.254.254172.25.254.135[root@lb01 conf]# curl 172.25.254.254172.25.254.134

关闭MASTER测试

[root@lb01 conf]# systemctl stop keepalived #这时VIP已经在BACKUP上,但是可以正常访问[root@lb01 conf]# curl 172.25.254.254172.25.254.135[root@lb01 conf]# curl 172.25.254.254172.25.254.134[root@lb01 conf]# curl 172.25.254.254172.25.254.135[root@lb01 conf]# curl 172.25.254.254172.25.254.134[root@lb01 conf]# curl 172.25.254.254172.25.254.135

使用主机,域名访问

172.25.254.254 #写hosts

正常访问

四、keepalived的其他特性

4.1 监控自动迁移脚本

keepalived解决的是主机级别的冗余,当nginx宕掉的时候,keepalive并不会迁移,这时VIP依然在该主机上,客户就不能访问到网站

使用脚本监控,当nginx挂掉,自动停掉keepalive,是VIP漂移,是业务不受影响

[root@lb01 conf]# mkdir /script

[root@lb01 conf]# vim /script/monitor.sh

#!/bin/bashwhile truedoif [ `ps -ef |grep nginx|grep -v grep |wc -l` -lt 2 ]thensystemctl stop keepalivedfisleep 5done

[root@lb01 conf]# cd /script/

[root@lb01 script]# chmod +x monitor.sh

[root@lb01 script]# /script/monitor.sh &

关掉nginx

[root@lb01 script]# nginx -s stop

VIP漂移到BACKUP上

[root@lb02 ~]# ip addr|grep 254.254

inet 172.25.254.254/24 scope global secondary ens33

访问:

[root@lb01 script]# curl 172.25.254.254172.25.254.134[root@lb01 script]# curl 172.25.254.254172.25.254.135[root@lb01 script]# curl 172.25.254.254172.25.254.134[root@lb01 script]# curl 172.25.254.254172.25.254.135

4.2 keepalive高可用脑裂脚本

[root@lb01 script]# ps -ef |grep monitorroot 80993 68563 0 07:04 pts/0 00:00:00 /bin/bash /script/monitor.sh[root@lb01 script]# kill -9 80993[1]+ Killed /script/monitor.sh[root@lb01 script]# ps -ef |grep monitorroot82773 68563 0 07:13 pts/0 00:00:00 grep --color=auto monitor[root@lb01 script]# systemctl restart keepalived

[root@lb02 ~]# mkdir /script

检测脑裂脚本,实现形式,当BACKUP可以ping通主,但是VIP依然在BACKUP,则认定为脑裂

[root@lb02 script]# vim check_split_brain.sh

#!/bin/bashwhile truedoping -c 2 -W 3 172.25.254.131 &>/dev/nullif [ $? -eq 0 -a `ip addr|grep 172.25.254.254|wc -l` -eq 1 ]thenecho "ha is split brain warning"elseecho "ha is OK"fisleep 3done

[root@lb02 ~]# sh /script/check_split_brain.sh

ha is OKha is OK[root@lb02 ~]# systemctl start firewalldha is split brain warningha is split brain warning[root@lb02 ~]# ip addr |grep 172.25.254.254inet 172.25.254.254/24 scope global secondary ens33[root@lb01 ~]# ip addr |grep 172.25.254.254inet 172.25.254.254/24 scope global secondary ens33[root@node4 ~]# systemctl stop firewalldha is OKha is OK

4.3 修改日志文件路径

配置文件默认在/var/log/messages

[root@lb01 ~]# tail -f /var/log/messagesApr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: VRRP_Instance(VI_1) Received advert with lower priority 100, ours 150, forcing new electionApr 13 07:41:26 node2 Keepalived_vrrp[82796]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:41:26 node2 Keepalived_vrrp[82796]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.25.254.254

修改日志文件位置

[root@lb01 ~]# vi /etc/sysconfig/keepalived

# Options for keepalived. See `keepalived --help' output and keepalived(8) and# keepalived.conf(5) man pages for a list of all options. Here are the most# common ones :## --vrrp-P Only run with VRRP subsystem.# --check -C Only run with Health-checker subsystem.# --dont-release-vrrp -V Dont remove VRRP VIPs & VROUTEs on daemon stop.# --dont-release-ipvs -I Dont remove IPVS topology on daemon stop.# --dump-conf-d Dump the configuration data.# --log-detail -D Detailed log messages.# --log-facility -S 0-7 Set local syslog facility (default=LOG_DAEMON)#KEEPALIVED_OPTIONS="-D -S 0 -d"

[root@lb01 ~]# vim /etc/rsyslog.conf

local0.* /var/log/keepalived.log

[root@lb01 ~]# systemctl restart rsyslog

[root@lb01 ~]# systemctl restart keepalived

[root@lb01 ~]# tail -f /var/log/keepalived.log

Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: VRRP_Instance(VI_1) Received advert with lower priority 100, ours 150, forcing new electionApr 13 07:46:23 node2 Keepalived_vrrp[84692]: Sending gratuitous ARP on ens33 for 172.25.254.254Apr 13 07:46:23 node2 Keepalived_vrrp[84692]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.25.254.254

4.4 keepalived多实例

[root@lb01 ~]# vim /etc/keepalived/keepalived.conf

1 ! Configuration File for keepalived2 3 global_defs {4 notification_email {5 283365585@6 }7 notification_email_from Alexandre.Cassen@firewall.loc8 smtp_server 192.168.200.19 smtp_connect_timeout 3010 router_id LVS_0111 vrrp_skip_check_adv_addr12 # vrrp_strict13 vrrp_garp_interval 014 vrrp_gna_interval 015 }16 17 vrrp_instance VI_1 {18state MASTER19interface ens3320virtual_router_id 5121priority 15022advert_int 123authentication {24 auth_type PASS25 auth_pass 111126}27virtual_ipaddress {28172.25.254.254/2429}30 }31 vrrp_instance VI_2 {32state BACKUP33interface ens3334virtual_router_id 5235priority 10036advert_int 137authentication {38 auth_type PASS39 auth_pass 111140}41virtual_ipaddress {42172.25.254.253/2443}44 }

[root@lb02 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalivedglobal_defs {notification_email {283365585@}notification_email_from Alexandre.Cassen@firewall.locsmtp_server 192.168.200.1smtp_connect_timeout 30router_id LVS_02vrrp_skip_check_adv_addr# vrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0}vrrp_instance VI_1 {state BACKUPinterface ens33virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.254/24}}vrrp_instance VI_2 {state MASTERinterface ens33virtual_router_id 52priority 150advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.253/24}}

检测

[root@lb01 ~]# systemctl restart keepalived[root@lb02 ~]# systemctl restart keepalived[root@lb01 ~]# ip addrinet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33valid_lft 499sec preferred_lft 499secinet 172.25.254.254/24 scope global secondary ens33valid_lft forever preferred_lft forever[root@lb02 ~]# ip addrinet 172.25.254.133/24 brd 172.25.254.255 scope global dynamic ens33valid_lft 422sec preferred_lft 422secinet 172.25.254.253/24 scope global secondary ens33valid_lft forever preferred_lft forever[root@lb02 ~]# systemctl stop keepalived[root@lb02 ~]# ip addr|grep 172.25.254inet 172.25.254.133/24 brd 172.25.254.255 scope global dynamic ens33[root@lb01 ~]# ip addr |grep 172.25.254inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33inet 172.25.254.254/24 scope global secondary ens33inet 172.25.254.253/24 scope global secondary ens33[root@lb02 ~]# systemctl start keepalived[root@lb02 ~]# ip addr|grep 172.25.254inet 172.25.254.133/24 brd 172.25.254.255 scope global dynamic ens33inet 172.25.254.253/24 scope global secondary ens33[root@lb01 ~]# ip addr |grep 172.25.254inet 172.25.254.131/24 brd 172.25.254.255 scope global dynamic ens33inet 172.25.254.254/24 scope global secondary ens33

参考:老男孩教育视频公开课/video/av25869969/?p=25

本内容不代表本网观点和政治立场,如有侵犯你的权益请联系我们处理。
网友评论
网友评论仅供其表达个人看法,并不表明网站立场。