关于mysql读写分离的一些测试

转载自 运维者 

MySQL Proxy是一个处于你的Client端和MySQL server端之间的简单程序,它可以监测、分析或改变它们的通信。它使用灵活,没有限制,常见的用途包括:负载平衡,故障、查询分析,查询过滤和修改等等。


MySQL-Proxy是处在你的MySQL数据库客户和服务端之间的程序,它还支持嵌入性脚本语言Lua。这个代理可以用来分析、监控和变换(transform)通信数据,它支持非常广泛的使用场景:

  1. 负载平衡和故障转移处理
  2. 查询分析和日志
  3. SQL宏(SQL macros)
  4. 查询重写(query rewriting)
  5. 执行shell命令

MySQL Proxy更强大的一项功能是实现“读写分离(Read/Write Splitting)”。基本的原理是让主数据库处理事务性查询,而从数据库处理SELECT查询。数据库复制被用来把事务性查询导致的变更同步到集群中的从数据库。

这些均来自互联网,已经熟的不能再熟了,也不是我要介绍的,下面开始我们的实战

环境:
就目前的环境来看我们已经搞会了master slave 以及 cluster等配置,也知道了变形虫(amoeba),它也一款代理软件,也支持读写分离,但是amoeba不支持事务,这点很蛋疼!所以我们这块,要继续研究一下mysql-proxy,因为它支持事务。

client—>proxy(读写分离)—–>cluster,是我们需要的环境了,为什么要这样,你懂的,percona cluster,所有的DB都支持读写,但是,如果写锁,那是非常郁闷的事!你能保证程序员不会给你来点刺激的事吗?所以我们要将读写进行分离,之前有用haproxy进行分离,但是存在一个问题,就是程序员写代码时,会不会严格的将两个端口或者两个IP写在代码里!一个人还好讲,一个团队,你能保证不会出问题,而且在国内的程序员的思维里,存在这种逻辑的估计很少,唉,运维就是这么苦B的职业,什么都要搞!下面我们测试测试,看看是不是能解决我们面临的问题!

一、安装mysql-proxy:

# yum list mysql-proxy
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.esocc.com
 * epel: ftp.jaist.ac.jp
 * extras: mirror.esocc.com
 * updates: centos.ustc.edu.cn
Available Packages
mysql-proxy.i686                                0.8.2-1.el6                            epel
mysql-proxy.x86_64                              0.8.2-1.el6                            epel

# yum install mysql-proxy
下面省略所有 ..........

多省事,不要老想着源码有多好管理,yum源就是方便!

二、配置启动需要的配置文件(vi /etc/sysconfig/mysql-proxy):

# Options for mysql-proxy 
ADMIN_USER="admin"
ADMIN_PASSWORD=""
ADMIN_LUA_SCRIPT="/usr/lib64/mysql-proxy/lua/admin.lua"
PROXY_USER="mysql-proxy"
PROXY_OPTIONS="--daemon --log-level=info --log-use-syslog"

原内容是这样的,我将其改为:

# Options for mysql-proxy 
ADMIN_USER="admin"
ADMIN_PASSWORD=""
ADMIN_LUA_SCRIPT="/usr/lib64/mysql-proxy/lua/admin.lua"
PROXY_USER="mysql-proxy"
PROXY_OPTIONS="--defaults-file=/etc/mysql-proxy.cnf"

mysql-proxy.cnf内容如下:

[mysql-proxy]
daemon = true
user = mysql-proxy
proxy-skip-profiling = true
keepalive = true
;max-open-files = 2048
event-threads = 50
pid-file = /var/run/mysql-proxy.pid
log-file = /var/log/mysql-proxy.log
log-level = debug
admin-address=:4401
admin-username=admin
admin-password=passw0rd
admin-lua-script=/usr/lib64/mysql-proxy/lua/admin.lua
proxy-address = 0.0.0.0:3307
proxy-backend-addresses = 192.168.1.64:3306
proxy-read-only-backend-addresses=192.168.1.66:3306, 192.168.1.244:3306
proxy-lua-script=/usr/lib64/mysql-proxy/lua/proxy/balance.lua

这里要多插两句,backend-addresses = 192.168.1.64:3306(读写) proxy-read-only-backend-address后面的DB主机只能是两台,超过两台就会有问题!如果说你有多个read-only-backend大可以分成两行写!

修改启动文件(vi /etc/init.d/mysql-proxy):

-    daemon $prog $PROXY_OPTIONS --pid-file=$PROXY_PID --user=$PROXY_USER --admin-username="$ADMIN_USER" --admin-lua-script="$ADMIN_LUA_SCRIPT" --admin-password="$ADMIN_PASSWORD"
+    daemon $prog $PROXY_OPTIONS --pid-file=$PROXY_PID

三、启动:

# /etc/init.d/mysql-proxy start
Starting mysql-proxy:                                      [  OK  ]

# tail -f /var/log/mysql-proxy.log
-06-20 23:08:17: (message) mysql-proxy 0.8.2 started
-06-20 23:08:17: (debug) max open file-descriptors = 1024
-06-20 23:08:17: (message) proxy listening on port 0.0.0.0:3307
-06-20 23:08:17: (message) added read/write backend: 192.168.1.64:3306
-06-20 23:08:17: (message) added read-only backend: 192.168.1.66:3306
-06-20 23:08:17: (message) added read-only backend: 192.168.1.244:3306
-06-20 23:08:17: (debug) now running as user: mysql (498/498)
-06-20 23:08:17: (message) chassis-event-thread.c:375: starting 49 threads

# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:3307                0.0.0.0:*                   LISTEN      8044/mysql-proxy    
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1035/sshd           
tcp        0      0 192.168.1.63:4505           0.0.0.0:*                   LISTEN      3509/python         
tcp        0      0 192.168.1.63:4506           0.0.0.0:*                   LISTEN      3501/python         
tcp        0      0 :::80                       :::*                        LISTEN      4442/httpd          
tcp        0      0 :::9200                     :::*                        LISTEN      12829/xinetd        
tcp        0      0 :::22                       :::*                        LISTEN      1035/sshd           
tcp        0      0 :::443                      :::*                        LISTEN      4442/httpd          
tcp        0      0 :::3306                     :::*                        LISTEN      6086/mysqld

端口3307已经成功启动

四、测试:

[root@YQD-intranet-salt-master sysconfig]# mysql -uroot -p123456 -h192.168.1.63 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.5.37-35.0-55 Percona XtraDB Cluster (GPL), Release rel35.0, Revision 756, WSREP version 25.10, wsrep_25.10.r3985

Copyright (c) 2009-2014 Percona LLC and/or its affiliates
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

关闭写服务器(192.168.1.64)这里出现大麻烦了,all backends are down:

# mysql -uroot -p123456 -h192.168.1.63 -P3307
Warning: Using a password on the command line interface can be insecure.
ERROR 1105 (HY000): (proxy) all backends are down

严格来讲,这里还是可以读的!!!为什么会这样

修改mysql-proxy.cnf的配置将lua脚本改为:

proxy-lua-script=/usr/lib64/mysql-proxy/lua/proxy/rw-splitting.lua

此时提示报错:

mysql> select count(id) from sbtest.sbtest;
+-----------+
| count(id) |
+-----------+
|   6000000 |
+-----------+
 row in set (0.95 sec)

mysql> show databases;
ERROR 2013 (HY000): Lost connection to MySQL server during query

mysql-proxy日志报错:
(critical) proxy-plugin.c.1127: I have no server backend, closing connection
-06-24 16:00:20: (critical) network-mysqld.c.1414: plugin_call(CON_STATE_READ_QUERY) failed
-06-24 16:00:20: (debug) [network-mysqld.c:937]: error on a connection (fd: -1 event: 0). closing client connection.

如果这样已经是读写分离了,那这样的一旦192.168.1.64挂了,连查看服务器都无法执行了,而且连续执行之后又有新问题,简单惨不忍睹:

mysql> show databases;
ERROR 1046 (3D000): No database selected
mysql> show databases;
ERROR 1046 (3D000): No database selected
mysql> show databases;
ERROR 1046 (3D000): No database selected
mysql> show databases;
ERROR 1046 (3D000): No database selected
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cat_db             |
| mysql              |
| performance_schema |
| sbtest             |
| test               |
+--------------------+
 rows in set (0.00 sec)

有时成功,有时不成功,OK,也许mysql-proxy只适合master-slave 对于cluster估计还得换方案:

OK,方案继续改进,即然已经测试到这个份上了,我们再启用haproxy,之前已经讲过,重复的部分我们就不讲了,这里增加一部分配置:

frontend yqd_all
bind *:33306
mode tcp
default_backend yqd_all

......

backend yqd_all
balance roundrobin
server read 127.0.0.1:3307 inter 12000 rise 3 fall 3
server write 127.0.0.1:3308 inter 12000 rise 3 fall 3

连接mysql的端口为:33306,haproxy启动3307和3308两个端口,分别是读和写,启动haproxy后,查看进程如下:

# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:3307                0.0.0.0:*                   LISTEN      27285/haproxy       
tcp        0      0 0.0.0.0:3308                0.0.0.0:*                   LISTEN      27285/haproxy       
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      27285/haproxy       
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1035/sshd           
tcp        0      0 192.168.1.63:4505           0.0.0.0:*                   LISTEN      3509/python         
tcp        0      0 0.0.0.0:33306               0.0.0.0:*                   LISTEN      27285/haproxy       
tcp        0      0 192.168.1.63:4506           0.0.0.0:*                   LISTEN      3501/python         
tcp        0      0 :::80                       :::*                        LISTEN      4442/httpd          
tcp        0      0 :::9200                     :::*                        LISTEN      12829/xinetd        
tcp        0      0 :::22                       :::*                        LISTEN      1035/sshd           
tcp        0      0 :::443                      :::*                        LISTEN      4442/httpd          
tcp        0      0 :::3306                     :::*                        LISTEN      6086/mysqld

此时一切变得正常了!

五、sysbench OLTP基准测试(mysql-proxy balance.lua的测试,已经不再使用):

OLTP基准测试模拟了一个简单的的事务处理系统的工作负载。下面的例子使用的是一张百万行记录的表,第一步先生成这张表:

# time sysbench --test=oltp --db-driver=mysql --mysql-engine-trx=yes --mysql-table-engine=innodb --mysql-host=192.168.1.63 --mysql-port=3307 --mysql-user=sbtest --mysql-password=sbpass --oltp-table-size=1000000 prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...

real	1m6.206s
user	0m0.221s
sys	0m0.042s

接下来可以运行测试,这个例子采用了16个并发线程,只读模拟测试:

# time sysbench --test=oltp --db-driver=mysql --mysql-engine-trx=yes --mysql-table-engine=innodb --mysql-host=192.168.1.63 --mysql-port=3307 --oltp-read-only=on --mysql-user=root --mysql-password=LVS071103 --oltp-table-size=1000000 --num-threads=16 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           0
        other:                           20000
        total:                           160000
    transactions:                        10000  (343.17 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 140000 (4804.35 per sec.)
    other operations:                    20000  (686.34 per sec.)

Test execution summary:
    total time:                          29.1403s
    total number of events:              10000
    total time taken by event execution: 466.1031
    per-request statistics:
         min:                                  3.58ms
         avg:                                 46.61ms
         max:                                540.16ms
         approx.  95 percentile:             147.98ms

Threads fairness:
    events (avg/stddev):           625.0000/18.04
    execution time (avg/stddev):   29.1314/0.00


real	0m29.884s
user	0m1.021s
sys	0m2.088s

去掉只读选项测试:

# sysbench --test=oltp --db-driver=mysql --mysql-engine-trx=yes --mysql-table-engine=innodb --mysql-host=192.168.1.63 --mysql-port=3307 --mysql-user=root --mysql-password=LVS071103 --oltp-table-size=10000 --num-threads=8 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140042
        write:                           50011
        other:                           20004
        total:                           210057
    transactions:                        10001  (70.73 per sec.)
    deadlocks:                           2      (0.01 per sec.)
    read/write requests:                 190053 (1344.04 per sec.)
    other operations:                    20004  (141.47 per sec.)

Test execution summary:
    total time:                          141.4043s
    total number of events:              10001
    total time taken by event execution: 1130.8833
    per-request statistics:
         min:                                  9.80ms
         avg:                                113.08ms
         max:                                610.27ms
         approx.  95 percentile:             303.89ms

Threads fairness:
    events (avg/stddev):           1250.1250/4.70
    execution time (avg/stddev):   141.3604/0.02

六、sysbench OLTP基准测试(haproxy的测试):

# time sysbench --test=oltp --db-driver=mysql --mysql-engine-trx=yes --mysql-table-engine=innodb --mysql-host=192.168.1.63 --mysql-port=33306 --oltp-read-only=on --mysql-user=root --mysql-password=LVS071103 --oltp-table-size=1000000 --num-threads=16 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           0
        other:                           20000
        total:                           160000
    transactions:                        10000  (1340.79 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 140000 (18771.05 per sec.)
    other operations:                    20000  (2681.58 per sec.)

Test execution summary:
    total time:                          7.4583s
    total number of events:              10000
    total time taken by event execution: 119.2205
    per-request statistics:
         min:                                  3.66ms
         avg:                                 11.92ms
         max:                                873.09ms
         approx.  95 percentile:              36.81ms

Threads fairness:
    events (avg/stddev):           625.0000/90.70
    execution time (avg/stddev):   7.4513/0.00


real	0m7.699s
user	0m0.839s
sys	0m2.423s

从这点上看时间上快了20来秒左右!

# sysbench --test=oltp --db-driver=mysql --mysql-engine-trx=yes --mysql-table-engine=innodb --mysql-host=192.168.1.63 --mysql-port=33306 --mysql-user=root --mysql-password=LVS071103 --oltp-table-size=10000000 --num-threads=16 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (247.60 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (4704.44 per sec.)
    other operations:                    20000  (495.20 per sec.)

Test execution summary:
    total time:                          40.3874s
    total number of events:              10000
    total time taken by event execution: 645.5854
    per-request statistics:
         min:                                  5.63ms
         avg:                                 64.56ms
         max:                               1190.69ms
         approx.  95 percentile:             239.82ms

Threads fairness:
    events (avg/stddev):           625.0000/149.41
    execution time (avg/stddev):   40.3491/0.02

使用haproxy仍然存在写冲突的问题,一旦冲突,就会写锁,真心不知道那些说使用的比较好的,他们是怎么做到的!

七,监控管理:
这里推荐两个clustercrontrol和haproxy本身的监控:

 

posted @ 2017-04-17 23:01  醉人  阅读(101)  评论(0)    收藏  举报