【mysql】不可不知的Metadata Lock

一、问题发生

说一个现象,当收到服务器报警之后,数据库服务器CPU使用超过90%,通过 show processlist 一看,满屏都是 Waiting for table metadata lock 状态的连接。第一反应就是kill掉这些连接,奈何连接实在太多,实在kill不过来,于是重启服务,重启果真能解决90%的问题,但如果不找到问题原因,问题也肯定会再次出现。

类似于下图(没找到Waiting for table metadata lock的示例,将就看下)

 Id	User	Host	db	Command	Time	State	Info	Memory_used	Memory_used_by_query	Logical_read	Physical_sync_read	Physical_async_read	Temp_user_table_size	Temp_sort_table_size	Temp_sort_file_size
1277733	data01	172.17.22.204:26173	birds	Query	50	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2275632	12912	1012951	1	0	0	0	0
1277751	data01	172.17.22.204:26745	birds	Query	49	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2245288	12912	936956	4	0	0	0	0
1277802	data01	172.17.22.204:29587	birds	Query	58	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2584048	12912	761473	2	0	0	0	0
1277825	data01	172.17.22.204:30637	birds	Query	57	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	1414176	12912	751966	0	0	0	0	0
1277827	data01	172.17.22.204:30639	birds	Query	56	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2497392	12912	757188	0	0	0	0	0
1277828	data01	172.17.22.204:30641	birds	Query	57	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	1134712	12912	739028	0	0	0	0	0
1277829	data01	172.17.22.204:30723	birds	Query	56	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	1134712	12912	725881	0	0	0	0	0
1277830	data01	172.17.22.204:30721	birds	Query	55	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2238288	12912	732720	0	0	0	0	0
1277831	data01	172.17.22.204:30765	birds	Query	56	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2497520	12912	757099	0	0	0	0	0
1277832	data01	172.17.22.204:30801	birds	Query	55	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	693261	0	0	0	0	0
1277879	data01	172.17.22.204:32905	birds	Query	34	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '300780'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	462365	0	0	0	0	0
1277880	data01	172.17.22.204:32907	birds	Query	33	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	425506	3	0	0	0	0
1277881	data01	172.17.22.204:32993	birds	Query	29	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '185711'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1138896	8544	483586	1	0	0	0	0
1277914	data01	172.17.22.204:34571	birds	Query	25	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2217736	12912	315580	0	0	0	0	0
1277915	data01	172.17.22.204:34573	birds	Query	0	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '557415'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1138640	8544	283988	0	0	0	0	0
1277916	data01	172.17.22.204:34677	birds	Query	25	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	309707	0	0	0	0	0
1277918	data01	172.17.22.204:34747	birds	Query	25	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	299223	0	0	0	0	0
1277919	data01	172.17.22.204:34805	birds	Sleep	0	NULL	1123104	8208	283855	3	0	0	0	0
1277920	data01	172.17.22.204:34829	birds	Query	6	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '512151'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	305759	5	0	0	0	0
1277922	data01	172.17.22.204:34909	birds	Query	20	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	253827	0	0	0	0	0
1277925	data01	172.17.22.204:35133	birds	Query	21	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	276768	0	0	0	0	0
1277929	data01	172.17.22.204:35319	birds	Query	20	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2217928	12912	251582	0	0	0	0	0
1277971	data01	172.17.22.204:36879	birds	Query	3	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '489209'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	45250	0	0	0	0	0
1277972	data01	172.17.22.204:36881	birds	Query	3	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '187583'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1150288	8544	40179	0	0	0	0	0
1277981	data01	172.17.22.204:37213	birds	Query	0	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '194152'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	7289	0	0	0	0	0
1277982	data01	172.17.22.204:37231	birds	Query	0	Writing to net	SELECT * FROM pre_forum_post WHERE tid=585078 AND position>=14361 AND position<14371 ORDER BY position	1164064	10280	374	0 0	0	0	0
1277984	data01	172.17.22.204:37285	NULL	Query	0	init	show full processlist	73800	8472	0	0	0	0	0	0

原因是什么:

MySQL在进行一些alter table等DDL操作时,如果该表上有未提交的事务则会出现 Waiting for table metadata lock ,而一旦出现metadata lock,该表上的后续操作都会被阻塞,由于当时直接把ALTER给kill了,所以基本没有现场

二、关于Table Metadata Lock

1、为什么会有 Metadata  Lock

源起于一个bug,MySQL官方文档链接:http://bugs.mysql.com/bug.php?id=989

Description:
If user1 has an active transaction on a table and then user2 drops this table, then user1 does COMMIT, then in the binlog we have something like:
DROP TABLE t;
BEGIN;
INSERT INTO t ... ;
COMMIT;
which is wrong.

 

这个bug大致意思是说:当一个会话在主库执行DML操作还没提交时,另一个会话对同一个对象执行了DDL操作如drop table,而由于MySQL的binlog是基于事务提交的先后顺序进行记录的,因此在从库上应用时,就出现了先drop table,然后再向table中insert的情况,导致从库应用出错。

因此,MySQL在5.5.3版本后引入了Metadata lock,只有在事务结束后才会释放Metadata lock,因此在事务提交或回滚前,是无法进行DDL操作的。

2、什么是Table Metadata Lock

首先,看看官方的说法,

MySQL官方文档位置:http://dev.mysql.com/doc/refman/5.6/en/metadata-locking.html

To ensure transaction serializability, the server must not permit one session to perform a data definition language (DDL) statement on a table that is used in an uncompleted explicitly or implicitly started transaction in another session.

The server achieves this by acquiring metadata locks on tables used within a transaction and deferring release of those locks until the transaction ends.

A metadata lock on a table prevents changes to the table's structure.

This locking approach has the implication that a table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.

从上面的描述可以看到,

  • MDL出现的初衷就是为了保护一个处于事务中的表的结构不被修改。

  • 这里提到的事务包括两类,显式事务和AC-NL-RO(auto-commit non-locking read-only)事务。显式事务包括两类:1. 关闭AutoCommit下的操作,2. 以begin或start transaction开始的操作。AC-NL-RO可理解为AutoCommit开启下的select操作。

  • MDL是事务级别的,只有在事务结束后才会释放。在此之前,其实也有类似的保护机制,只不过是语句级别的。

需要注意的是,MDL不仅仅适用于表,同样也适用于其它对象,如下表所示,其中,"等待状态"对应的是"show processlist"中的State。

 

为了提高数据库的并发度,MDL被细分为了11种类型。

  • MDL_INTENTION_EXCLUSIVE

  • MDL_SHARED

  • MDL_SHARED_HIGH_PRIO

  • MDL_SHARED_READ

  • MDL_SHARED_WRITE

  • MDL_SHARED_WRITE_LOW_PRIO

  • MDL_SHARED_UPGRADABLE

  • MDL_SHARED_READ_ONLY

  • MDL_SHARED_NO_WRITE

  • MDL_SHARED_NO_READ_WRITE

  • MDL_EXCLUSIVE

常用的有MDL_SHARED_READ,MDL_SHARE D_WRITE及MDL_EXCLUSIVE,其分别用于SELECT操作,DML操作及DDL操作。其它类型的对应操作可参考源码sql/mdl.h。

对于MDL_EXCLUSIVE,官方的解释是, 

An exclusive metadata lock.

A connection holding this lock can modify both table's metadata and data.

No other type of metadata lock can be granted while this lock is held.

To be used for CREATE/DROP/RENAME TABLE statements and for execution of certain phases of other DDL statements.

简而言之,MDL_EXCLUSIVE是独占锁,在其持有期间是不允许其它类型的MDL被授予,自然也包括SELECT和DML操作。

这也就是为什么DDL操作被阻塞时,后续其它操作也会被阻塞。

关于MDL的补充

1. MDL的最大等待时间由lock_wait_timeout参数决定,其默认值为31536000(365天)。在使用工具进行DDL操作时,这个值就不太合理。事实上,pt-online-schema-change和gh-ost对其就进行了相应的调整,其中,前者60s,后者3s。

2. 如果一个SQL语法上有效,但执行时报错,如,列名不存在,其同样会获取MDL锁,直到事务结束才释放。

三、Metadata Lock解决了什么问题

MySQL 从5.5.3版本,对Metadata lock进行了调整,主要是MDL锁持有的周期从语句变成了事务, 其原因主要是解决两个问题:

  • 问题1: 破坏事务隔离级别 在repeatable read的隔离级别下,多次的select语句执行过程中,会因为其它session的DDL语句,而导致select语句执行的结果不相同,破坏了RR的隔离级别。

  • 问题2: 破坏binlog的顺序 在对表的DML过程中,会因为其它session的DDL语句,导致binlog里的event顺序在备库执行的结果和主库不一致。

从MySQL 5.5.3开始,MDL锁的持有周期变成了事务,解决了上面提到的两个问题,但在autocommit=off的情况下,也大大增加了阻塞的可能性。DBA对于阻塞的case,处理起来又比较麻烦,原因就是MDL锁的阻塞情况没有暴露明确的信息。

从MySQL 5.7.6开始,可以通过performance schema来查询MDL锁的持有情况。

RR事务隔离级别下不可重复读的问题,演示环境,MySQL 5.5.0。

session1> begin;
Query OK, 0 rows affected (0.00 sec)
session1> select * from t1;
+------+------+
| id  | name |
+------+------+
|    1 | a    |
|    2 | b    |
+------+------+
2 rows in set (0.00 sec)
session2> alter table t1 add c1 int;
Query OK, 2 rows affected (0.02 sec)
Records: 2  Duplicates: 0  Warnings: 0
session1> select * from t1;
Empty set (0.00 sec)
session1> commit;
Query OK, 0 rows affected (0.00 sec)
session1> select * from t1;
+------+------+------+
| id  | name | c1  |
+------+------+------+
|    1 | a    | NULL |
|    2 | b    | NULL |
+------+------+------+
2 rows in set (0.00 sec)

可以看到,虽然是RR隔离级别,但在开启事务的情况下,第二次查询却没有结果

主从复制问题

包括主从数据不一致,主从复制中断等。 如下面的主从数据不一致。

session1> create table t1(id int,name varchar(10)) engine=innodb;
Query OK, 0 rows affected (0.00 sec)
session1> begin;
Query OK, 0 rows affected (0.00 sec)
session1> insert into t1 values(1,'a');
Query OK, 1 row affected (0.00 sec)
session2> truncate table t1;
Query OK, 0 rows affected (0.46 sec)
session1> commit;
Query OK, 0 rows affected (0.35 sec)
session1> select * from t1;
Empty set (0.00 sec)

再来看看从库的结果

session1> select * from t1;
+------+------+------+
| id   | name | c1   |
+------+------+------+
|    1 | a    | NULL |
+------+------+------+
1 row in set (0.00 sec)

看看binlog的内容,可以看到,truncate操作记录在前,insert操作记录在后。

# at 7140
#180714 19:32:14 server id 1  end_log_pos 7261    Query    thread_id=31    exec_time=0    error_code=0
SET TIMESTAMP=1531567934/*!*/;
create table t1(id int,name varchar(10)) engine=innodb
/*!*/;
# at 7261
#180714 19:32:30 server id 1  end_log_pos 7333    Query    thread_id=32    exec_time=0    error_code=0
SET TIMESTAMP=1531567950/*!*/;
BEGIN
/*!*/;
# at 7333
#180714 19:32:30 server id 1  end_log_pos 7417    Query    thread_id=32    exec_time=0    error_code=0
SET TIMESTAMP=1531567950/*!*/;
truncate table t1
/*!*/;
# at 7417
#180714 19:32:30 server id 1  end_log_pos 7444    Xid = 422
COMMIT/*!*/;
# at 7444
#180714 19:32:34 server id 1  end_log_pos 7516    Query    thread_id=31    exec_time=0    error_code=0
SET TIMESTAMP=1531567954/*!*/;
BEGIN
/*!*/;
# at 7516
#180714 19:32:24 server id 1  end_log_pos 7611    Query    thread_id=31    exec_time=0    error_code=0
SET TIMESTAMP=1531567944/*!*/;
insert into t1 values(1,'a')
/*!*/;
# at 7611
#180714 19:32:34 server id 1  end_log_pos 7638    Xid = 421
COMMIT/*!*/;

如果会话2执行的是drop table操作,还会导致主从中断。

有意思的是,如果会话2执行的是alter table操作,其依旧会被阻塞,阻塞时间受innodb_lock_wait_timeout参数限制。

mysql> show processlist;
+----+------+-----------+----------+---------+------+-------------------+---------------------------+
| Id | User | Host      | db       | Command | Time | State             | Info                      |
+----+------+-----------+----------+---------+------+-------------------+---------------------------+
| 54 | root | localhost | NULL     | Query   |    0 | NULL              | show processlist          |
| 58 | root | localhost | slowtech | Sleep   | 1062 |                   | NULL                      |
| 60 | root | localhost | slowtech | Query   |   11 | copy to tmp table | alter table t1 add c1 int |
+----+------+-----------+----------+---------+------+-------------------+---------------------------+
3 rows in set (0.00 sec)

可以看到没有MDL锁的情况下,会出现以上两种问题,当有MDL锁保护之后,就正常了

InnoDB层已经有了IS、IX这样的意向锁,有同学觉得可以用来实现上述例子的并发控制。但由于MySQL是Server-Engine架构,所以MDL锁是在Server中实现。另外,MDL锁还能实现其他粒度级别的锁,比如全局锁、库级别的锁、表空间级别的锁,这是InnoDB存储引擎层不能直接实现的锁。

但与InnoDB锁的实现一样,MDL锁也是类似对一颗树的各个对象从上至下进行加锁(对树进行加锁具体见:《MySQL技术内幕:InnoDB存储引擎》)。但是MDL锁对象的层次更多,简单来看有如下的层次:

上图中显示了最常见的4种MDL锁的对象,并且注明了常见的SQL语句会触发的锁。与InnoDB层类似的是,某些类型的MDL锁会从上往下一层层进行加锁。比如LOCK TABLE … WRITE这样的SQL语句,其首先会对GLOBAL级别加INTENTION_EXCLUSIVE锁,再对SCHEMA级别加INTENTION_EXCLUSIVE锁,最后对TABLE级别加SHARED_NO_READ_WRITE锁。这里最令人意外的是还有COMMIT对象层次的锁,其实这主要用于XA事务中。比如分布式事务已经PREPARE成功,但是在XA COMMIT之前有其他会话执行了FLUSH TABLES WITH READ LOCK这样的操作,那么分布式事务的提交就需要等待。

除了上图标注的对象,其实还有TABLESPACE、FUNCTION、PROCEDURE、EVENT等其他对象类型,其实都是为了进行并发控制。只是这些在MySQL数据库中都不常用,故不再赘述(当然也是为了偷懒)。

GLOBAL=0,TABLESPACE,SCHEMA,TABLE,FUNCTION,PROCEDURE,TRIGGER,EVENT,COMMIT,USER_LEVEL_LOCK,LOCKING_SERVICE,NAMESPACE_END 

目前MDL有如下锁模式,锁之间的兼容性可见源码mdl.cc:

锁模式 对应SQL
MDL_INTENTION_EXCLUSIVE GLOBAL对象、SCHEMA对象操作会加此锁
MDL_SHARED FLUSH TABLES with READ LOCK
MDL_SHARED_HIGH_PRIO 仅对MyISAM存储引擎有效
MDL_SHARED_READ SELECT查询
MDL_SHARED_WRITE DML语句
MDL_SHARED_WRITE_LOW_PRIO 仅对MyISAM存储引擎有效
MDL_SHARED_UPGRADABLE ALTER TABLE
MDL_SHARED_READ_ONLY LOCK xxx READ
MDL_SHARED_NO_WRITE FLUSH TABLES xxx,yyy,zzz READ
MDL_SHARED_NO_READ_WRITE FLUSH TABLE xxx WRITE
MDL_EXCLUSIVE ALTER TABLE xxx PARTITION BY …

MDL锁的性能与并发改进

讲到这同学们会发现MDL锁的开销并不比InnoDB层的行锁要小,而且这可能是一个更为密集的并发瓶颈。MySQL 5.6和5.5版本通常通过调整如下两个参数来进行并发调优:

  • metadata_locks_cache_size: MDL锁的缓存大小

  • metadata_locks_hash_instances:通过分片来提高并发度,与InnoDB AHI类似

MySQL 5.7 MDL锁的最大改进之处在于将MDL锁的机制通过lock free算法来实现,从而提高了在多核并发下数据库的整体性能提升。

MDL锁的诊断

MySQL 5.7版本之前并没有提供一个方便的途径来查看MDL锁,github上有一名为mysql-plugin-mdl-info的项目,通过插件的方式来查看,非常有想法的实现,大赞。好在官方也意识到了这个问题,于是在MySQL 5.7中的performance_schea库下新增了一张表metadata_locks,用其来查看MDL锁那是相当的方便:

不过默认PS并没有打开此功能,需要手工将wait/lock/metadata/sql/mdl监控给打开

SELECT * FROM performance_schema.setup_instruments;  
UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME ='global_instrumentation';
UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME ='wait/lock/metadata/sql/mdl';
select * from performance_schema.metadata_locks\G

  

参考文章

http://mysql.taobao.org/monthly/2015/10/02/

http://www.yunweipai.com/archives/19818.html

https://www.jianshu.com/p/683bd7fcb7a0

https://www.cnblogs.com/zengkefu/p/5690385.html

https://www.cnblogs.com/cchust/p/3826398.html

https://www.cnblogs.com/xpchild/p/3789068.html

http://www.ywnds.com/?p=7209&viewuser=42

https://mp.weixin.qq.com/s?__biz=MzU3ODU4MTgyOQ==&mid=2247483729&idx=1&sn=03711c826fcef68ea530a9cfa4decb25&chksm=fd72657bca05ec6d356040139c0f649800b5d5c454a4d675241fdd6a8809bb1657778a4aa032&scene=21#wechat_redirect

https://mp.weixin.qq.com/s?__biz=MzU3ODU4MTgyOQ==&mid=2247483735&idx=1&sn=a3de9f2632581b4643781b44cdbfe819&chksm=fd72657dca05ec6b1a82338e8416b58b58278daee749620393ec268a98833e6d2b0b25fe20ff&mpshare=1&scene=1&srcid=0913ujxgOYxYZzV6fuLfAMhn#rd

http://www.ywnds.com/?p=7209&viewuser=42

https://www.cnblogs.com/zengkefu/p/5690385.html

https://m.aliyun.com/yunqi/articles/51031

https://www.cnblogs.com/xpchild/p/3790139.html

posted @ 2018-09-20 14:18  踏雪无痕SS  阅读(7480)  评论(0编辑  收藏  举报