代码改变世界

multipath多路径实验02-配置多路径软件

2017-07-19 22:53  AlfredZhao  阅读(...)  评论(...编辑  收藏

multipath多路径实验02-配置多路径软件

在上一篇文章《multipath多路径实验01-构建iSCSI模拟环境》,我构建了iSCSI的模拟环境,在文章最后,已经成功配置并在主机上认出多个链路分别映射出的盘。而本文我就要对这样的多路径环境,通过Linux自带multipath多路径软件聚合成可以方便我们后期使用的形式。

1.安装多路径软件包

查询是否安装了多路径软件包:
rpm -qa |grep device-mapper-multipath

如果没有安装,则用yum安装即可:
yum install device-mapper-multipath

[root@jyrac1 ~]# yum install device-mapper-multipath
Loaded plugins: product-id, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package device-mapper-multipath.x86_64 0:0.4.9-72.el6 will be installed
--> Processing Dependency: device-mapper-multipath-libs = 0.4.9-72.el6 for package: device-mapper-multipath-0.4.9-72.el6.x86_64
--> Processing Dependency: libmultipath.so()(64bit) for package: device-mapper-multipath-0.4.9-72.el6.x86_64
--> Processing Dependency: libmpathpersist.so.0()(64bit) for package: device-mapper-multipath-0.4.9-72.el6.x86_64
--> Running transaction check
---> Package device-mapper-multipath-libs.x86_64 0:0.4.9-72.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================================
 Package                                   Arch                Version                     Repository        Size
==================================================================================================================
Installing:
 device-mapper-multipath                   x86_64              0.4.9-72.el6                ISO              116 k
Installing for dependencies:
 device-mapper-multipath-libs              x86_64              0.4.9-72.el6                ISO              180 k

Transaction Summary
==================================================================================================================
Install       2 Package(s)

Total download size: 297 k
Installed size: 653 k
Is this ok [y/N]: y
Downloading Packages:
------------------------------------------------------------------------------------------------------------------
Total                                                                             2.8 MB/s | 297 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : device-mapper-multipath-libs-0.4.9-72.el6.x86_64                                               1/2 
  Installing : device-mapper-multipath-0.4.9-72.el6.x86_64                                                    2/2 
  Verifying  : device-mapper-multipath-libs-0.4.9-72.el6.x86_64                                               1/2 
  Verifying  : device-mapper-multipath-0.4.9-72.el6.x86_64                                                    2/2 

Installed:
  device-mapper-multipath.x86_64 0:0.4.9-72.el6                                                                   

Dependency Installed:
  device-mapper-multipath-libs.x86_64 0:0.4.9-72.el6                                                              

Complete!

如果不方便配置yum,也可以直接rpm安装这两个包:

[root@jyrac2 Packages]# rpm -ivh device-mapper-multipath-libs-0.4.9-72.el6.x86_64.rpm
warning: device-mapper-multipath-libs-0.4.9-72.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                ########################################### [100%]
   1:device-mapper-multipath########################################### [100%]
[root@jyrac2 Packages]# rpm -ivh device-mapper-multipath-0.4.9-72.el6.x86_64.rpm
warning: device-mapper-multipath-0.4.9-72.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                ########################################### [100%]
   1:device-mapper-multipath########################################### [100%]

2.设置服务开机启动

查询当前开机自启用设置:
chkconfig --list|grep multipathd

确认内核成功加载:

modprobe -l |grep multipath     

确认开启开机自启动:

[root@jyrac1 ~]# chkconfig multipathd on
[root@jyrac1 ~]# chkconfig --list|grep multipathd
multipathd      0:off   1:off   2:on    3:on    4:on    5:on    6:off

3.生成multipath配置文件

生成multipath配置文件:

/sbin/mpathconf  --enable

确认服务状态:

service multipathd status

4.multipath的常用命令

常用命令:
--生成multipath配置文件
/sbin/mpathconf --enable

--显示多路径的布局
multipath -ll

--重新刷取
multipath -v2

--清空所有多路径
multipath -F

针对上述常用命令,实际操作的记录,供参考:

[root@jyrac1 ~]# multipath -ll
Jul 18 15:57:18 | DM multipath kernel driver not loaded
Jul 18 15:57:18 | /etc/multipath.conf does not exist, blacklisting all devices.
Jul 18 15:57:18 | A sample multipath.conf file is located at
Jul 18 15:57:18 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Jul 18 15:57:18 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Jul 18 15:57:18 | DM multipath kernel driver not loaded
[root@jyrac1 ~]# chkconfig --list|grep multipathd
multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@jyrac1 ~]# modprobe -l |grep multipath   
kernel/drivers/md/dm-multipath.ko
[root@jyrac1 ~]# service multipathd status
multipathd is stopped
[root@jyrac1 ~]# service multipathd start
Starting multipathd daemon: [  OK  ]
[root@jyrac1 ~]# multipath -ll
Jul 18 15:59:00 | /etc/multipath.conf does not exist, blacklisting all devices.
Jul 18 15:59:00 | A sample multipath.conf file is located at
Jul 18 15:59:00 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Jul 18 15:59:00 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
[root@jyrac1 ~]# /sbin/mpathconf --enable

[root@jyrac1 ~]# multipath -v2
Jul 18 16:02:52 | mpatha: ignoring map
Jul 18 16:02:52 | mpathb: ignoring map
Jul 18 16:02:52 | mpathc: ignoring map
Jul 18 16:02:52 | mpathd: ignoring map
Jul 18 16:02:52 | mpathe: ignoring map
Jul 18 16:02:52 | mpathf: ignoring map
create: mpathg (1IET     00010001) undef IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=1 status=undef
| `- 10:0:0:1 sdi 8:128 undef ready  running
`-+- policy='round-robin 0' prio=1 status=undef
  `- 11:0:0:1 sdj 8:144 undef ready  running
[root@jyrac1 ~]# multipath -ll
mpathg (1IET     00010001) dm-2 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 10:0:0:1 sdi 8:128 active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 11:0:0:1 sdj 8:144 active ready  running
[root@jyrac1 ~]# 

这里测试没有问题后,为了后续方便测试ASM磁盘组,在iSCSI服务端又分了一个10g的lv,模拟存储端分出的一个LUN,最后在这边就应该有两个多路径聚合盘了。同样的方法在RAC的另一个节点也都配置好。

5.udev绑定权限

我这里系统是RHEL6.5,对于multipath的权限,手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。 搜索对应的配置文件模板:
[root@jyrac1 ~]# find / -name 12-*
/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules

根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面:

vi /etc/udev/rules.d/12-dm-permissions.rules
# MULTIPATH DEVICES
#
# Set permissions for all multipath devices
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"

查看多路径对应的底层dm设备:

[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1
brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2
brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-3

--启动start_udev
[root@jyrac2 rules.d]# start_udev 
Starting udev: [  OK  ]

--查看权限:
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk     253, 0 Jul 19 21:35 /dev/dm-0
brw-rw---- 1 root disk     253, 1 Jul 19 21:35 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3

--等30s后再查,权限固定:
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk     253, 0 Jul 19 21:35 /dev/dm-0
brw-rw---- 1 root disk     253, 1 Jul 19 21:35 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3

发现权限固定不变没有问题了,在RAC的另一个节点也同样设置。
这时多路径的信息再查看下:

[root@jyrac1 ~]# multipath -ll
mpathi (1IET     00010002) dm-2 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:2 sdk 8:160 active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 12:0:0:2 sdl 8:176 active ready  running
mpathg (1IET     00010001) dm-3 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:1 sdi 8:128 active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 12:0:0:1 sdj 8:144 active ready  running
[root@jyrac1 ~]# 

在另一个节点也要达到同样效果。

6.新建ASM磁盘组测试

使用grid用户登录xmanager图形界面调出asmca,创建一个新的磁盘组data2,使用这两个盘。冗余度选择Normal冗余: 最终加磁盘组成功,对应查询磁盘组和磁盘的相关信息如下:
SQL> select group_number, disk_number, name, path, mode_status, voting_file  from v$asm_disk order by 1,2;

GROUP_NUMBER DISK_NUMBER NAME                           PATH                                               MODE_ST V
------------ ----------- ------------------------------ -------------------------------------------------- ------- -
           1           0 DATA1_0000                     /dev/asm-diske                                     ONLINE  N
           1           1 DATA1_0001                     /dev/asm-diskf                                     ONLINE  N
           1           2 DATA1_0002                     /dev/asm-diskg                                     ONLINE  N
           2           0 FRA1_0000                      /dev/asm-diskh                                     ONLINE  N
           3           0 OCR1_0000                      /dev/asm-diskb                                     ONLINE  Y
           3           1 OCR1_0001                      /dev/asm-diskc                                     ONLINE  Y
           3           2 OCR1_0002                      /dev/asm-diskd                                     ONLINE  Y
           4           0 DATA2_0000                     /dev/mapper/mpathg                                 ONLINE  N
           4           1 DATA2_0001                     /dev/mapper/mpathi                                 ONLINE  N

9 rows selected.

SQL> select group_number, name, total_mb, free_mb, usable_file_mb, offline_disks, state, type from v$asm_diskgroup;

GROUP_NUMBER NAME                             TOTAL_MB    FREE_MB USABLE_FILE_MB OFFLINE_DISKS STATE       TYPE
------------ ------------------------------ ---------- ---------- -------------- ------------- ----------- ------
           1 DATA1                               15360      13204          13204             0 CONNECTED   EXTERN
           2 FRA1                                 5120       3773           3773             0 CONNECTED   EXTERN
           3 OCR1                                 3072       2146            561             0 MOUNTED     NORMAL
           4 DATA2                               20480      20290          10145             0 MOUNTED     NORMAL

至此,有关Linux自带的multipath多路径软件相关的实验已经全部完成。