Mycat高可用集群搭建

HaProxy+Keepalived+Mycat高可用集群配置

 

部署图：

  

 

集群部署图理解：

1、keepalived和haproxy必须装在同一台机器上（如192.168.46.161机器上，keepalived和haproxy都要安装），keepalived负责为该服务器抢占vip（虚拟ip），抢占到vip后，对该主机的访问可以通过原来的ip（192.168.46.161）访问，也可以直接通过vip（192.168.46.180）访问。

2、192.168.46.162上的keepalived也会去抢占vip，抢占vip时有优先级，配置keepalived.conf中的（priority 150 #数值愈大，优先级越高,192.168.46.162上改为120，master和slave上该值配置不同）决定。但是一般哪台主机上的keepalived服务先启动就会抢占到vip，即使是slave，只要先启动也能抢到。

3、haproxy负责将对vip的请求分发到mycat上。起到负载均衡的作用，同时haproxy也能检测到mycat是否存活，haproxy只会将请求转发到存活的mycat上。

4、如果一台服务器（keepalived+haproxy服务器）宕机，另外一台上的keepalived会立刻抢占vip并接管服务。如果一台mycat服务器宕机，haporxy转发时不会转发到宕机的mycat上，所以mycat依然可用。

 

1、Haproxy的安装

1.1、配置haprxoy

（下载地址: http://www.haproxy.org/#down）

useradd haproxy

cd haproxy-1.4.27/

make TARGET=linux26 PREFIX=/usr/local/haproxy ARCH=x86_64

make install PREFIX=/usr/local/haproxy

cd /usr/local/haproxy

vi haproxy.cfg

增加如下内容：

global

log 127.0.0.1 local0

        maxconn 4096

        chroot /usr/local/haproxy

        user haproxy

        group haproxy

        daemon

defaults

        log global

        option dontlognull

        retries 3

        option redispatch

        maxconn 2000

        contimeout 5000

        clitimeout 50000

        srvtimeout 50000

listen admin_stats 192.168.46.180:48800

        stats uri /admin-status

        stats auth admin:admin

        mode http

        option httplog

listen mycat_service 192.168.46.180:18066

        mode tcp

        option tcplog

          option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www

        balance roundrobin

          server mycat_161 192.168.46.161:8066 check port 48700 inter 5s rise 2 fall 3

          server mycat_162 192.168.46.162:8066 check port 48700 inter 5s rise 2 fall 3

        srvtimeout 20000

listen mycat_admin 192.168.46.180:19066

        mode tcp

        option tcplog

          option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www

        balance roundrobin

          server mycat_161 192.168.46.161:9066 check port 48700 inter 5s rise 2 fall 3

          server mycat_162 192.168.46.162:9066 check port 48700 inter 5s rise 2 fall 3

        srvtimeout 20000

        

1.2、配置haproxy记录日志功能

yum install rsyslog -y

cd /etc/rsyslog.d/

vi haproxy.conf

增加内容：

$ModLoad imudp

$UDPServerRun 514

local0.* /var/log/haproxy.log

 

vi /etc/rsyslog.conf

在#### RULES ####上面一行加入以下内容

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

在local7.* /var/log/boot.log下面加入以下内容

local0.* /var/log/haproxy.log

 

重启rsyslog服务并将rsyslog加入自动启动服务

service rsyslog restart

chkconfig --add rsyslog

chkconfig --level 2345 rsyslog on

 

1.3 配置监听mycat是否存活(在Mycat server1 Mycat server2上都需要添加检测端口48700的脚本，为此需要用到xinetd，xinetd为linux系统的基础服务)

yum install xinetd -y

cd /etc/xinetd.d

vi mycat_status

增加内容：

service mycat_status

{

        flags = REUSE

        socket_type = stream

        port = 48700

        wait = no

        user = nobody

        server = /usr/local/bin/mycat_status

        log_on_failure += USERID

        disable = no

}

 

vi /usr/local/bin/mycat_status (创建xinetd启动服务脚本)

增加内容：

 

#!/bin/bash

#/usr/local/bin/mycat_status.sh

# This script checks if a mycat server is healthy running on localhost. It will

# return:

#

# "HTTP/1.x 200 OK\r" (if mycat is running smoothly)

#

# "HTTP/1.x 503 Internal Server Error\r" (else)

mycat=`/usr/local/mycat/bin/mycat status |grep 'not running'| wc -l`

if [ "$mycat" = "0" ];

then

/bin/echo -e "HTTP/1.1 200 OK\r\n"

else

/bin/echo -e "HTTP/1.1 503 Service Unavailable\r\n"

fi

 

修改脚本文件权限

chmod 777 /usr/local/bin/mycat_status

chmod 777 /etc/xinetd.d/mycat_status

 

将启动脚本加入服务

vi /etc/services

末尾增加：

mycat_status 48700/tcp # mycat_status

 

重启xinetd服务并将xinetd加入自启动服务

service xinetd restart

chkconfig --add xinetd

chkconfig --level 2345 xinetd on

 

验证mycat_status服务是否启动成功

netstat -antup|grep 48700

 

1.4、创建haproxy启停脚本

 

 1.4.1、启动脚本

  vi  /usr/local/haproxy/sbin/start

增加内容：

#!/bin/sh

/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg &

  增加权限：

chmod +x /usr/local/haproxy/sbin/start

 

 1.4.2、关闭脚本

  vi /usr/local/haproxy/sbin/stop

  增加内容：

#!/bin/sh

ps -ef | grep sbin/haproxy | grep -v grep |awk '{print $2}'|xargs kill -s 9

  增加权限：

chmod +x /usr/local/haproxy/sbin/stop

 

 1.4.3、授权

  chown -R haproxy.haproxy /usr/local/haproxy/*

 

 

1.5、启动haproxy

启动haproxy前必须先启动keepalived，否则启动不了。

启动命令：

/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

启动haproxy异常情况

如果报以下错误：

[root@localhost bin]# /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

[ALERT] 183/115915 (12890) :Starting proxy admin_status: cannot bind socket

[ALERT] 183/115915 (12890) :Starting proxy allmycat_service: cannot bind socket

[ALERT] 183/115915 (12890) :Starting proxy allmycat_admin: cannot bind socket

原因为：该机器没有抢占到vip，如果另一台服务启动正常，这个错误可以忽略不管，如果另一台也一样，使用ping vip命令看看vip是否生效，如果没有生效，说明keepalived没有启动成功，回去检查keepalived的异常再说。启动后可以通过http://192.168.46.180:48800/admin-status (用户名密码都是admin，haproxy.cfg中配置的)

 

 

2、Keepalived安装

 

2.1 openssl安装

(下载地址：https://www.openssl.org/source/)

  openssl必须安装，否则安装keepalived时无法编译，keepalived依赖openssl

 

  tar -zxvf openssl-1.0.2l.tar.gz

  cd openssl-1.0.2l

  ./config --prefix=/usr/local/openssl

  ./config -t

make depend

make

make test

make install

ln -s /usr/local/openssl /usr/local/ssl

 

vi /etc/ld.so.conf

在文件末尾加入以下内容

/usr/local/openssl/lib

 

修改环境变量

vi /etc/profile

在文件末尾加入以下内容

export OPENSSL=/usr/local/openssl/bin

export PATH=$PATH:$OPENSSL

 

source /etc/profile

安装openssl-devel

yum install openssl-devel -y

 

测试

ldd /usr/local/openssl/bin/openssl

linux-vdso.so.1 => (0x00007fff996b9000)

libdl.so.2 =>/lib64/libdl.so.2 (0x00000030efc00000)

libc.so.6 =>/lib64/libc.so.6 (0x00000030f0000000)

/lib64/ld-linux-x86-64.so.2 (0x00000030ef800000)

 

which openssl

/usr/bin/openssl

openssl version

OpenSSL 1.0.0-fips 29 Mar 2010

 

 

2.2 keepalived安装

在192.168.46.161，192.168.46.162两台机器进行keepalived安装

tar zxvf keepalived-1.2.13.tar.gz

cd keepalived-1.2.13

./configure --prefix=/usr/local/keepalived

make

make install

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/

mkdir /etc/keepalived

cd /etc/keepalived/

cp /usr/local/keepalived/etc/keepalived/keepalived.conf  /etc/keepalived

mkdir -p /usr/local/keepalived/var/log

 

keepalived配置

建检查haproxy是否存活的脚本

mkdir /etc/keepalived/scripts

cd /etc/keepalived/scripts

 

vi /etc/keepalived/keepalived.conf

 

 

Master的配置：

! Configuration File for keepalived

vrrp_script chk_http_port {

        script"/etc/keepalived/scripts/check_haproxy.sh"

        interval 2

        weight 2

}

vrrp_instance VI_1 {

        state MASTER

        interface eth1

        virtual_router_id 51

        priority 150

        advert_int 1

        authentication

        auth_type PASS

        auth_pass 1111

}

track_script {

        chk_http_port

}

virtual_ipaddress {

        192.168.46.180 dev eth1 scope global

}

        notify_master /etc/keepalived/scripts/haproxy_master.sh

        notify_backup /etc/keepalived/scripts/haproxy_backup.sh

        notify_fault /etc/keepalived/scripts/haproxy_fault.sh

        notify_stop /etc/keepalived/scripts/haproxy_stop.sh

}

 

 

Slave的配置：

! Configuration File for keepalived

vrrp_script chk_http_port {

        script"/etc/keepalived/scripts/check_haproxy.sh"

        interval 2

        weight 2

}

vrrp_instance VI_1 {

        state BACKUP

        interface eth1

        virtual_router_id 51

        priority 150

        advert_int 1

        authentication

        auth_type PASS

        auth_pass 1111

}

track_script {

        chk_http_port

}

virtual_ipaddress {

        192.168.46.180 dev eth1 scope global

}

        notify_master /etc/keepalived/scripts/haproxy_master.sh

        notify_backup /etc/keepalived/scripts/haproxy_backup.sh

        notify_fault /etc/keepalived/scripts/haproxy_fault.sh

        notify_stop /etc/keepalived/scripts/haproxy_stop.sh

}

 

1. virtual_router_id 51 这个代表一个集群组，如果同一个网段还有另一组集群，请使用不同的组编号区分。如换成52、53等。

2. interface eth1 和172.17.210.103 dev eth1 scope global中的eth1指的是网卡，如果是多网卡，可能会有

 eth0，eth1，eth2…，可以使用ifconfig命令查看，确保eth0是本机存在的网卡地址。有些服务器如果只有一个网卡，但被人为把eth0改成eth1了，你再写eth0就找不到了的。

 

 

vi /etc/keepalived/scripts/check_haproxy.sh

 

#!/bin/bash

STARTHAPROXY="/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg"

STOPKEEPALIVED="/etc/init.d/keepalived stop"

LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log"

echo "[check_haproxy status]" >> $LOGFILE

A=`ps -C haproxy --no-header |wc -l`

echo "[check_haproxy status]" >> $LOGFILE

date >> $LOGFILE

if [ $A -eq 0 ];then

echo $STARTHAPROXY >> $LOGFILE

$STARTHAPROXY >> $LOGFILE 2>&1

sleep 5

fi

if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then

exit 0

else

exit 1

fi

 

 

vi /etc/keepalived/scripts/haproxy_master.sh

 

#!/bin/bash

STARTHAPROXY=`/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg`

STOPHAPROXY=`ps -ef | grep sbin/haproxy | grep -v grep | awk '{print $2}'| xargs kill -s 9`

LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log"

echo "[master]" >> $LOGFILE

date >> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "stop haproxy...." >> $LOGFILE 2>&1

$STOPHAPROXY >> $LOGFILE 2>&1

echo "start haproxy...." >> $LOGFILE 2>&1

$STARTHAPROXY >> $LOGFILE 2>&1

echo "haproxy stared ..." >> $LOGFILE

 

vi /etc/keepalived/scripts/haproxy_backup.sh

 

#!/bin/bash

STARTHAPROXY=`/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg`

STOPHAPROXY=`ps -ef | grep sbin/haproxy | grep -v grep | awk '{print $2}'| xargs kill -s 9`

LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log"

echo "[backup]" >> $LOGFILE

date >> $LOGFILE

echo "Being backup...." >> $LOGFILE 2>&1

echo "stop haproxy...." >> $LOGFILE 2>&1

$STOPHAPROXY >> $LOGFILE 2>&1

echo "start haproxy...." >> $LOGFILE 2>&1

$STARTHAPROXY >> $LOGFILE 2>&1

echo "haproxy stared ..." >> $LOGFILE

 

vi /etc/keepalived/scripts/haproxy_fault.sh

 

#!/bin/bash

LOGFILE=/usr/local/keepalived/var/log/keepalived-haproxy-state.log

echo "[fault]" >> $LOGFILE

date >> $LOGFILE

 

vi /etc/keepalived/scripts/haproxy_stop.sh

 

#!/bin/bash

LOGFILE=/usr/local/keepalived/var/log/keepalived-haproxy-state.log

echo "[stop]" >> $LOGFILE

date >> $LOGFILE

 

赋予脚本可执行权限

chmod 777 /etc/keepalived/scripts/*

 

将keepalived加入自启动服务并启动

chkconfig --add keepalived

     chkconfig --level 2345 keepalived on

service keepalived start

 

3、搭建完成

 

 

表明搭建完成！