RH7.5 linux系统环境下挂载MD存储

====================================

Linux MPIO Lab:

Env: VM have 2 iscsi NIC(eth1,eth2)

1. Mount RHEL source ISO,install "iscsi-initiator" package.

#mount /dev/sr0 /mnt

#vi /etc/yum.repos.d/local.repo

[local]

name=local

baseurl=file:///mnt/Server

enabled=1

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

#yum clean all

#yum repolist

#yum -y install iscsi-initiator-utils

enable iscsi service when boot

#chkconfig sendmail off //turn off sendmail service,or pls set full DNS name for linux host!

#chkconfig iscsi on

#service iscsi start

change sysct.conf       （EQ才需要配置，因为网口网段一样的，回来的包不知道从哪里走）

#vi /etc/sysctl.conf

net.ipv4.conf.all.arp_ignore=1 //https://access.redhat.com/knowledge/solutions/24141

   or:

net.ipv4.conf.eth0.rp_filter = 0

net.ipv4.conf.eth0.arp_ignore = 1

net.ipv4.conf.eth0.arp_announce = 2

net.ipv4.conf.eth1.rp_filter = 0

net.ipv4.conf.eth1.arp_ignore = 1

net.ipv4.conf.eth1.arp_announce = 2

#2: Loose mode as defined in RFC3704 Loose Reverse Path. Each incoming packet's source address is also tested against the forwarding table and if the source address is not reachable via any interface the packet check will fail.

IF iSCSI failover time is taking too long,need setting parameter in iscsid.conf!

#/etc/iscsi/iscsid.conf: //https://access.redhat.com/site/solutions/386613

node.conn[0].timeo.noop_out_interval = 5

node.conn[0].timeo.noop_out_timeout = 5

node.session.timeo.replacement_timeout = 5

#Multipath :

Reduce the polling interval from 10 seconds (default) to 5 seconds. This number can grow to 4 * polling_interval, so a difference of 40s vs 20s

Reduce the checker_timeout (uses default scsi timeout) :

defaults { 

user_friendly_names yes

polling_interval 5

checker_timeout 15

} 

2. iscsi initiator config

Add 2 iface and bind to physical nic       (将物理网卡和虚拟网卡绑定，如果不配置iface物理网卡故障不会自动飘到另一个网卡，通过配置虚拟网卡实现，windows不用这样配置，因为windows底层（看不见的东西）已经配置好了)

#rm -rf /var/lib/iscsi/ifaces/

#iscsiadm -m iface -I eth1 -o new    （eth1为虚拟网卡）

#iscsiadm -m iface -I eth2 -o new

#iscsiadm -m iface -I eth1 --op=update -n iface.net_ifacename -v eth1（前面eth1为虚拟网卡，后面eth1为物理卡）

#iscsiadm -m iface -I eth2 --op=update -n iface.net_ifacename -v eth2

#ls -l /var/lib/iscsi/ifaces/

discovery MD iscsi virtual ports through iface nic.(需要使用server配置的所有iface虚拟网卡 dicovery对应MD的所有同网段in 口 ip，最新的FW只需要discovery 一次，不需要都discovery)

#iscsiadm -m discovery -t st -p 192.168.60.228 -I eth1

192.168.60.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

192.168.61.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

192.168.60.229:3260,2 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

192.168.61.229:3260,2 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

#iscsiadm -m discovery -t st -p 192.168.61.228 -I eth2

192.168.60.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

192.168.61.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

192.168.60.229:3260,2 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

192.168.61.229:3260,2 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

3.Login to MDSM

#use MDSM create vd and regeister linux host;mapping MD's vd to linux

4.Login to MD's VD (需要使用server配置的所有iface虚拟网卡 login对应MD的所有同网段in 口 ip)（这个跟window区别，windows只要connect（login）一次，然后在properties加入其它路径（session））

#iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0 -p 192.168.60.228:3260 -l -I eth1

Logging in to [iface: eth1, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.60.228,3260] (multiple)

Logging in to [iface: eth1, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.60.228,3260] (multiple)

Login to [iface: eth1, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.60.228,3260] successful.

Login to [iface: eth1, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.60.228,3260] successful.

ser

#iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0 -p 192.168.61.228:3260 -l -I eth2

Logging in to [iface: eth2, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.61.228,3260] (multiple)

Logging in to [iface: eth2, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.61.228,3260] (multiple)

Login to [iface: eth2, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.61.228,3260] successful.

Login to [iface: eth2, target: iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0, portal: 192.168.61.228,3260] successful.

#iscsiadm -m node --login

or

#iscsiadm -m node -T iqn... -p <group ip>:3260 -l

#iscsiadm -m session   （查看到4个login）

tcp: [5] 192.168.60.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

tcp: [6] 192.168.60.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

tcp: [7] 192.168.61.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

tcp: [8] 192.168.61.228:3260,1 iqn.1984-05.com.dell:powervault.md3600i.6782bcb00015540200000000502ad8d0

#service iscsi restart

4. install multipath&config ser

install multipath daemon and lsscsi utility.

#yum -y install device-mapper-multipath

#yum -y install lsscsi

#lsscsi //check MD's lun have 4 multipath(/dev/sdb,c,d,e)!!

[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0

[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda

[7:0:0:0]    disk    DELL     MD36xxi          0784  /dev/sdb

[7:0:0:31]   disk    DELL     Universal Xport  0784  -

[8:0:0:0]    disk    DELL     MD36xxi          0784  /dev/sdc

[8:0:0:31]   disk    DELL     Universal Xport  0784  -

[9:0:0:0]    disk    DELL     MD36xxi          0784  /dev/sdd

[9:0:0:31]   disk    DELL     Universal Xport  0784  -

[10:0:0:0]   disk    DELL     MD36xxi          0784  /dev/sde

[10:0:0:31]  disk    DELL     Universal Xport  0784  -

config multipath.conf:

#cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/

#vi /etc/multipath.conf 

blacklist {

            device {

                        vendor "*"

                        product "Universal Xport" //blacklist MDxxi's access lun!!!

            }

            ….

            }

devices {

        device {

                vendor                  "DELL"

                product                 "(MD32xxi|MD36xxi)"

                path_grouping_policy    group_by_prio

                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"

                path_checker            rdac

                path_selector           "round-robin 0"

                hardware_handler        "1 rdac"

                failback                immediate

                no_path_retry           -1

                features                "2 pg_init_retries 50"

                rr_min_io               10

        }

}

or Generate a default multipath.conf file:

# mpathconf --enable --user_friendly_names y --with_module y --with_multipathd y --with_chkc

start multipath daemon:

#service multipathd start

#chkconfig multipathd on

#chkconfig lvm2-monitor off //https://access.redhat.com/knowledge/solutions/59483

check CML vol muitipath and policy:

#multipath -ll -v2

Mar 21 08:25:09 | multipath.conf line 92, invalid keyword: prio_callout

mpathb (36782bcb0001430660000e08c514b5b8d) dm-2 DELL,MD36xxi //mpathb is logical device for MD36xxi's physical device(/dev/sdb,c,d,e)

size=10G features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 7:0:0:0  sdb 8:16 active ready running

  |- 8:0:0:0  sdc 8:32 active ready running

  |- 9:0:0:0  sdd 8:48 active ready running

  `- 10:0:0:0 sde 8:64 active ready running

# ls -l /dev/mapper/mpathb

lrwxrwxrwx. 1 root root 7 Mar 21 08:24 /dev/mapper/mpathb -> ../dm-2

change MD36xxi's WWID(36782bcb0001430660000e08c514b5b8d) to alias name:demo-vol!!

#vi /etc/multipath.conf

multipath use alias name:

defaults {

        user_friendly_names yes

}

multipaths {

        multipath {

                wwid                    36782bcb0001430660000e08c514b5b8d

                alias                   demo-vol

}

}

#service multipathd restart

#multipath -ll -v2

Mar 21 08:31:36 | multipath.conf line 92, invalid keyword: prio_callout

demo-vol (36782bcb0001430660000e08c514b5b8d) dm-2 DELL,MD36xxi //logical device name /dev/mpathb change to user define alias:demo-vol!!

size=10G features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 7:0:0:0  sdb 8:16 active ready running

  |- 8:0:0:0  sdc 8:32 active ready running

  |- 9:0:0:0  sdd 8:48 active ready running

  `- 10:0:0:0 sde 8:64 active ready running

#ls -l /dev/mapper/demo-vol

lrwxrwxrwx. 1 root root 7 Mar 21 08:31 /dev/mapper/demo-vol -> ../dm-2

mkfs

#fdisk /dev/mapper/demo-vol

#partprobe

#kpartx -a /dev/mapper/demo-vol

#mkfs.ext3 /dev/mapper/demo-volp1

#mount /dev/mapper/demo-volp1 /mnt

#vi /etc/fstab

/dev/mapper/demo-volp1  /mnt                    ext3    _netdev         0 0 //add mount point and parameter:_netdev for iscsi device!!

#mount -a

#df -H

Filesystem             Size   Used  Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root

5.9G   2.4G   3.3G  42% /

tmpfs                  523M      0   523M   0% /dev/shm

/dev/sda1              508M    49M   433M  11% /boot

/dev/mapper/demo-volp1

11G   158M   9.9G   2% /mnt

#reboot

MPIO test

#dd if=/dev/zero of=test bs=1M count=100 //#Test use dd write 100M file

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.37705 s, 278 MB/s

#ls -l

total 102520

drwx------. 2 root root     16384 Mar 21 08:36 lost+found

-rw-r--r--. 1 root root 104857600 Mar 21 08:47 test

#ifdown eth2 //simulator eth2 iscsi nic failed

#multipath -ll -v2

Mar 21 08:47:53 | multipath.conf line 92, invalid keyword: prio_callout

Mar 21 08:49:54 | rdac checker failed to set TAS bit

Mar 21 08:49:54 | rdac checker failed to set TAS bit

demo-vol (36782bcb0001430660000e08c514b5b8d) dm-2 DELL,MD36xxi

size=10G features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 7:0:0:0  sdb 8:16 active ready  running

  |- 8:0:0:0  sdc 8:32 active ready  running

  |- 9:0:0:0  sdd 8:48 failed faulty running //lost 2 physical device!

  `- 10:0:0:0 sde 8:64 failed faulty running

#dd if=/dev/zero of=test bs=1M count=100 //lost 2 physical device(/dev/sdd,sde) still can write data!

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.335314 s, 313 MB/s

#ifup eth2 //physical iscsi nic restore

#ping -I eth2 192.168.61.228

PING 192.168.61.228 (192.168.61.228) from 192.168.61.206 eth2: 56(84) bytes of data.

64 bytes from 192.168.61.228: icmp_seq=1 ttl=64 time=0.453 ms

...

--- 192.168.61.228 ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 3469ms

rtt min/avg/max/mdev = 0.448/0.461/0.490/0.022 ms

# multipath -ll

Mar 21 08:54:06 | multipath.conf line 92, invalid keyword: prio_callout

demo-vol (36782bcb0001430660000e08c514b5b8d) dm-2 DELL,MD36xxi

size=10G features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 7:0:0:0  sdb 8:16 active ready running

  |- 8:0:0:0  sdc 8:32 active ready running

  |- 9:0:0:0  sdd 8:48 active ready running //2 physical device name return to normal!!

  `- 10:0:0:0 sde 8:64 active ready running

5.delete iscsi setting

#iscsiadm -m node -T iqn.2002-03.com.compellent:5000d31000ed1326 -p 172.16.170.90:3260 -u

#iscsiadm -m node -T iqn.2002-03.com.compellent:5000d31000ed1327 -p 172.16.170.90:3260 -u

#iscsiadm -m node -o delete -T iqn.2002-03.com.compellent:5000d31000ed1326 --portal 172.16.170.90

#iscsiadm -m node -o delete -T iqn.2002-03.com.compellent:5000d31000ed1327 --portal 172.16.170.90

#iscsiadm -m session

#iscsiadm -m iface -o delete -I eth1

#iscsiadm -m iface -o delete -I eth2

#iscsiadm -m iface