keepalived+nginx双机热备+负载均衡
最近因业务扩展，需要将当前的apache 转为nginx(web), 再在web前端放置nginx(负载均衡)。同时结合keepalived 对前端nginx实现HA。 nginx进程基于于Master+Slave(worker)多进程模型，自身具有非常稳定的子进程管理功能。在Master进程分配模式下，Master进程永远不进行业务处理，只是进行任务分发，从而达到Master进程的存活高可靠性，Slave(worker)进程所有的业务信号都 由主进程发出，Slave(worker)进程所有的超时任务都会被Master中止，属于非阻塞式任务模型。 Keepalived是Linux下面实现VRRP 备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。二者结合，可以构架出比较稳定的软件lb方案。
准备4台电脑来做这个实验:
192.168.232.132 web服务器 192.168.232.133 web服务器 192.168.232.134 keepalived nginx 192.168.232.135 keepalived nginx
虚拟IP (VIP)：192.168.232.16
134\135两个主机配置虚拟IP
下面以135为例：
vi /etc/sysconfig/network-scripts/ifcfg-eth2:0
[plain]view plaincopy print? DEVICE=eth2:0TYPE=EthernetONBOOT=yesBOOTPROTO=staticDNS1=192.168.232.2IPADDR=192.168.232.16NETMASK=255.255.255.0GETWAY=192.168.232.2
DEVICE=eth2:0TYPE=EthernetONBOOT=yesBOOTPROTO=staticDNS1=192.168.232.2IPADDR=192.168.232.16NETMASK=255.255.255.0GETWAY=192.168.232.2
service network restart
使用ifconfig查看效果：
[plain]view plaincopy print? eth2Linkencap:EthernetHWaddr00:0C:29:49:90:5Binetaddr:192.168.232.135Bcast:192.168.232.255Mask:255.255.255.0inet6addr:fe80::20c:29ff:fe49:905b/64Scope:LinkUPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1RXpackets:66322errors:0dropped:0overruns:0frame:0TXpackets:31860errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:1000RXbytes:67624991(64.4MiB)TXbytes:2723877(2.5MiB)Interrupt:19Baseaddress:0x2000eth2:0Linkencap:EthernetHWaddr00:0C:29:49:90:5Binetaddr:192.168.232.16Bcast:192.168.232.255Mask:255.255.255.0UPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1Interrupt:19Baseaddress:0x2000loLinkencap:LocalLoopbackinetaddr:127.0.0.1Mask:255.0.0.0inet6addr:::1/128Scope:HostUPLOOPBACKRUNNINGMTU:16436Metric:1RXpackets:22622errors:0dropped:0overruns:0frame:0TXpackets:22622errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:1236328(1.1MiB)TXbytes:1236328(1.1MiB)
eth2Link encap:Ethernet HWaddr 00:0C:29:49:90:5B inet addr:192.168.232.135 Bcast:192.168.232.255 Mask:255.255.255.0inet6 addr: fe80::20c:29ff:fe49:905b/64 Scope:LinkUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:66322 errors:0 dropped:0 overruns:0 frame:0TX packets:31860 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:67624991 (64.4 MiB) TX bytes:2723877 (2.5 MiB)Interrupt:19 Base address:0x2000eth2:0 Link encap:Ethernet HWaddr 00:0C:29:49:90:5B inet addr:192.168.232.16 Bcast:192.168.232.255 Mask:255.255.255.0UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1Interrupt:19 Base address:0x2000lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0inet6 addr: ::1/128 Scope:HostUP LOOPBACK RUNNING MTU:16436 Metric:1RX packets:22622 errors:0 dropped:0 overruns:0 frame:0TX packets:22622 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:0RX bytes:1236328 (1.1 MiB) TX bytes:1236328 (1.1 MiB)
说明生效了。
134\135两个主机安装keepalived和nginx
nginx安装：
1、导入外部软件库 rpm -Uvh /pub/ius/stable/Redhat/6/i386/epel-release-6-5.noarch.rpm rpm -Uvh /pub/ius/stable/Redhat/6/i386/ius-release-1.0-10.ius.el6.noarch.rpm rpm -Uvh /packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm 以下添加注释 mirrorlist=/mirrorlist?repo=ius-el6&arch=$basearch 以下删除注释 #baseurl=/pub/ius/stable/Redhat/5/$basearch 2、yum安装nginx yum install nginx
keepalived安装：
安装依赖 yum -y install gcc gcc+ gcc-c++ yum install popt-devel openssl openssl-devel libssl-dev libnl-devel popt-devel
安装内核 yum -y install kernel kernel-devel 当前kernel代码建立连接 ln -s /usr/src/kerners/2.6....../ /usr/src/linux
安装keepalived wget /software/keepalived-1.2.2.tar.gz tar -zxvf keepalived-1.2.2.tar.gz cd keepalived-1.2.2 ./configure make make install
拷贝相应的文件
cp /usr/local/sbin/keepalived /usr/sbin/ cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/ cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/ cp -r /usr/local/etc/keepalived/ /etc/
配置keeplived和nginx主机
134/135执行都执行以下操作： vi /etc/nginx/conf.d/default.conf
[plain]view plaincopy print? server{listen8088;server_namelocalhost;location/{root/var/www/html;indexindex.htmlindex.htm;}error_page500502503504/50x.html;location=/50x.html{root/usr/share/nginx/html;}}
server {listen 8088;server_name localhost;location / {root /var/www/html;index index.html index.htm;}error_page 500 502 503 504 /50x.html;location = /50x.html {root /usr/share/nginx/html;}}
135执行以下操作： vi /var/www/html/index.html
[plain]view plaincopy print? <html><head><title>Welcometonginx!</title></head><bodybgcolor="white"text="black"><center><h1>Welcometonginx!192.168.232.135</h1></center></body></html>
<html> <head> <title>Welcome to nginx!</title> </head> <body bgcolor="white" text="black"> <center><h1>Welcome to nginx! 192.168.232.135</h1></center> </body> </html>
134执行以下操作： vi /var/www/html/index.html
[plain]view plaincopy print? <html><head><title>Welcometonginx!</title></head><bodybgcolor="white"text="black"><center><h1>Welcometonginx!192.168.232.134</h1></center></body></html>
<html> <head> <title>Welcome to nginx!</title> </head> <body bgcolor="white" text="black"> <center><h1>Welcome to nginx! 192.168.232.134</h1></center> </body> </html>
134执行以下操作： vi /etc/keepalived/keepalived.conf
[plain]view plaincopy print? !ConfigurationFileforkeepalivedglobal_defs{notification_email{#acassen@firewall.loc#failover@firewall.loc#sysadmin@firewall.loc}#notification_email_fromAlexandre.Cassen@firewall.loc#smtp_server192.168.200.1#smtp_connect_timeout30router_idLVS_DEVEL}vrrp_scriptchk_http_port{script"</dev/tcp/127.0.0.1/8088"interval1weight-2}vrrp_instanceVI_1{stateMASTERinterfaceeth2virtual_router_id51priority100advert_int1authentication{auth_typePASSauth_pass1111}virtual_ipaddress{192.168.232.16}track_script{chk_http_port}}
! Configuration File for keepalivedglobal_defs {notification_email {#acassen@firewall.loc#failover@firewall.loc#sysadmin@firewall.loc}#notification_email_from Alexandre.Cassen@firewall.loc#smtp_server 192.168.200.1#smtp_connect_timeout 30router_id LVS_DEVEL}vrrp_script chk_http_port {script "</dev/tcp/127.0.0.1/8088"interval 1weight -2}vrrp_instance VI_1 {state MASTERinterface eth2virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.232.16}track_script {chk_http_port}}
135执行以下操作： vi /etc/keepalived/keepalived.conf
[plain]view plaincopy print? !ConfigurationFileforkeepalivedglobal_defs{notification_email{#acassen@firewall.loc#failover@firewall.loc#sysadmin@firewall.loc}#notification_email_fromAlexandre.Cassen@firewall.loc#smtp_server192.168.200.1#smtp_connect_timeout30router_idLVS_DEVEL}vrrp_scriptchk_http_port{script"</dev/tcp/127.0.0.1/8088"interval1weight-2}vrrp_instanceVI_1{stateBACKUPinterfaceeth2virtual_router_id51priority99advert_int1authentication{auth_typePASSauth_pass1111}virtual_ipaddress{192.168.232.16}track_script{chk_http_port}}
! Configuration File for keepalivedglobal_defs {notification_email {#acassen@firewall.loc#failover@firewall.loc#sysadmin@firewall.loc}#notification_email_from Alexandre.Cassen@firewall.loc#smtp_server 192.168.200.1#smtp_connect_timeout 30router_id LVS_DEVEL}vrrp_script chk_http_port {script "</dev/tcp/127.0.0.1/8088"interval 1weight -2}vrrp_instance VI_1 {state BACKUPinterface eth2virtual_router_id 51priority 99advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.232.16}track_script {chk_http_port}}
Tips：
state 参数值：主的是MASTER、备用的是BACKUP priority 参数值： MASTER > BACKUP virtual_router_id： 参数值要一样
测试测试：
两台测试机134\135均启动keepalived和nginx service keepalived restart service keepalived nginx
验证nginx启动正常： 访问 master：http://192.168.232.134:8088/ 访问 backup: http://192.168.232.135:8088/
查看keepalived的日志信息：
134\135均打开日志信息方便查看keepalived动态： tail -f /var/log/messages
浏览器打开虚拟ip访问：http://192.168.232.16:8080/ ，此时显示IP为192.168.232.134
服务器层的双机热备（比如服务器宕机、keepalived宕了）测试：
kill 192.168.232.134（master) 的keepalived进程 killall keepalived 134的日志信息如下：
[plain]view plaincopy print? Jun1118:03:10localhostKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth1for192.168.232.16Jun1118:03:15localhostKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth1for192.168.232.16Jun1119:30:44localhostKeepalived:TerminatingonsignalJun1119:30:44localhostKeepalived:StoppingKeepalivedv1.2.2(06/10,)Jun1119:30:44localhostKeepalived_vrrp:TerminatingVRRPchildprocessonsignalJun1119:30:44localhostKeepalived_healthcheckers:TerminatingHealthcheckerchildprocessonsignal
Jun 11 18:03:10 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.232.16Jun 11 18:03:15 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.232.16Jun 11 19:30:44 localhost Keepalived: Terminating on signalJun 11 19:30:44 localhost Keepalived: Stopping Keepalived v1.2.2 (06/10,)Jun 11 19:30:44 localhost Keepalived_vrrp: Terminating VRRP child process on signalJun 11 19:30:44 localhost Keepalived_healthcheckers: Terminating Healthchecker child process on signal
135的日志信息如下：
[plain]view plaincopy print? Jun1119:30:50localhostKeepalived_vrrp:VRRP_Instance(VI_1)settingprotocolVIPs.Jun1119:30:50localhostKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth2for192.168.232.16Jun1119:30:50localhostKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.232.16addedJun1119:30:55localhostKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth2for192.168.232.16
Jun 11 19:30:50 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.Jun 11 19:30:50 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.232.16Jun 11 19:30:50 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.232.16 addedJun 11 19:30:55 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.232.16
刷新http://192.168.232.16:8080/ ， 此时显示IP为192.168.232.135。
再次启动192.168.232.134的keepalived进程，192.168.232.134会自动接管成为master，192.168.232.135自动转为backup，从测试结果看，备机能成功接管，已经实现了热备。
应用层（web）的双机热备（比如nginx进程被意外kill、web端口不通）试验：
关闭192.168.232.134（master) 的nginx服务： service nginx stop
134的日志信息如下：
[plain]view plaincopy print? Jun1119:38:49localhostKeepalived_vrrp:VRRP_Script(chk_http_port)failedJun1119:38:51localhostKeepalived_vrrp:VRRP_Instance(VI_1)ReceivedhigherprioadvertJun1119:38:51localhostKeepalived_vrrp:VRRP_Instance(VI_1)EnteringBACKUPSTATEJun1119:38:51localhostKeepalived_vrrp:VRRP_Instance(VI_1)removingprotocolVIPs.Jun1119:38:51localhostKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.232.16removed
Jun 11 19:38:49 localhost Keepalived_vrrp: VRRP_Script(chk_http_port) failedJun 11 19:38:51 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advertJun 11 19:38:51 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATEJun 11 19:38:51 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.Jun 11 19:38:51 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.232.16 removed
135的日志信息如下：
[plain]view plaincopy print? Jun1119:38:52localhostKeepalived_vrrp:VRRP_Instance(VI_1)forcinganewMASTERelectionJun1119:38:53localhostKeepalived_vrrp:VRRP_Instance(VI_1)TransitiontoMASTERSTATEJun1119:38:54localhostKeepalived_vrrp:VRRP_Instance(VI_1)EnteringMASTERSTATEJun1119:38:54localhostKeepalived_vrrp:VRRP_Instance(VI_1)settingprotocolVIPs.Jun1119:38:54localhostKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth2for192.168.232.16Jun1119:38:54localhostKeepalived_healthcheckers:NetlinkreflectorreportsIP192.168.232.16addedJun1119:38:59localhostKeepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPsoneth2for192.168.232.16
Jun 11 19:38:52 localhost Keepalived_vrrp: VRRP_Instance(VI_1) forcing a new MASTER electionJun 11 19:38:53 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATEJun 11 19:38:54 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATEJun 11 19:38:54 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.Jun 11 19:38:54 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.232.16Jun 11 19:38:54 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.232.16 addedJun 11 19:38:59 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.232.16
刷新http://192.168.232.16:8080/ ， 此时显示IP为192.168.232.135。
再次启动192.168.232.134的nginx进程，192.168.232.134会自动接管成为master，192.168.232.135自动转为backup，从测试结果看，备机能成功接管，已经实现了热备。
为什么主备的参数state都是MASTER，对的你没有看错确实要都设置成一样的，不然并不能实现我们想要的VIP漂浮的效果，我测试很久才发现的.state都设置成MASTER后，会根据priority的值大小竞争来决定谁是真正的MASTER，脚本检测也是在失败的时候会把权重减去相应的值，比如原来master（181）的priority=100，如果脚本检测到端口8088无法连接，就会priority-2=98，< S-B（150）的priority（99），此时 S-B（150） 将竞争成为master，这样就实现了web应用的热备。
如果以上实验都没有问题了，那么就该nginx负载均衡的配置了，配置修改参见如下：/e421083458/article/details/30086413
