openfile学习笔记
Openfiler是在rPath Linux基础上开发的，它能够作为一个独立的Linux操作系统发行。Openfiler是一款非常好的存储管理操作系统，开源免费，通过web界面对
存储磁盘的管理，支持现在流行的网络存储技术IP-SAN和NAS，支持iSCSI（Internet Small Computer System Interface, 学名ISCSI HBA）、NFS、SMB/CIFS及FTP等
协议。
一． 安装openfiler
先下载一个Openfiler 软件，然后安装到我们的虚拟机上。 Openfiler是基于Linux的存储平台，安装过程和安装一般的Linux系统一样。
下载地址：http://www.openfiler.com/community/download/
安装的第一个界面：
这里有一个磁盘的配置。 我选择了手动配置。 我给openfiler 是40g 的磁盘空间。 系统占2G，交换区1G。 剩下的空间没有分配。
安装完成之后的界面如下:
在这里有提示我们通过web进行访问。 并且提示了访问的地址：
Https://192.168.1.1:446/. 默认账户是openfiler密码为password. 我们登陆之后可以修改用户的密码。
二． 存储端（target）配置
Openfiler 的配置，可以参考Oracle 的这遍文档：
http://www.oracle.com/technology/global/cn/pub/articles/hunter_rac10gr2_iscsi.html#9
http://www.oracle.com/technetwork/cn/articles/hunter-rac11gr2-iscsi-083834-zhs.html#11
2.1 启动iscsi target 服务
在Service 里面启动iscsi target。 启动之后，下次重启会自动启该服务。
2.2 配置iscsi initiator 访问IP
只有配置了IP 才有权限访问openfiler 存储。在system 选项的最下面有配置选项，把IP 写上即可。 注意这里的子网掩码，写的是255.255.255.255
2.3 创建卷设备
现在我们来配置共享设备。 先对我们没有格式的分区格式化成扩展分区，一定要扩展分区：
[root@san ~]# fdisk /dev/sda
The number of cylinders for this disk is set to 5221.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Partition number (1-4): 3
First cylinder (383-5221, default 383):
Using default value 383
Last cylinder or +size or +sizeM or +sizeK (383-5221, default 5221):
Using default value 5221
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@san ~]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 255 2048256 83 Linux
/dev/sda2 256 382 1020127+ 82 Linux swap / Solaris
/dev/sda3 383 5221 38869267+ 5 Extended
格式化之后，我们在openfiler的网页中就能看到这个磁盘信息，如果不格式化，或者格式化错误，是无法编辑的。
页面网下拉，我们能看到创建分区：
把所有空间全部创建成一个分区，这个就是一个卷。 之后窗口会显示：
创建完成后， 选择volume Groups。
然后输入vg名称和对应的设备，确定即可。
至此，我们已经创建完成了一个叫san的卷组。 但是我们在环境中使用的是卷。 所以我们还需要在这个卷组里创建它的卷。
点击旁边的add volume选项：
在这个页面往下拉，我们可以看到创建卷的选项：
这里我把所有的空间都创建到一个逻辑卷里。
逻辑卷创建完成以后，我们需要创建一个iscsi target， 然后把逻辑卷和这个target 映射起来，这样就可以通过这个target 与服务器进行连接。 点机iSCSI
Target，创建Target IQN
选择LUN Mapping, 将ISCSI 和 逻辑卷对应起来
配置能够访问逻辑卷的Network ACL 权限，这个ip 是在system 的选项里设置的。 这个之前已经配置过。 这里可以设置多个IP, 可以控制哪个IP 允许访问哪个逻辑卷。从而可以多个用户同时使用存储而互不影响。
至此， 存储的服务端已经配置完成。 在这一步，我们创建了一个逻辑卷然后与ISCSI target 进行了对应。 客户端的服务器就通过这个ISCSI target 进行连接。
Openfiler target的配置文件是: /etc/ietd.conf。
[root@san etc]# cat /etc/ietd.conf
##### WARNING!!! - This configuration file generated by Openfiler. DO NOT MANUALLY EDIT. #####
Target iqn.2006-01.com.san
HeaderDigest None
DataDigest None
MaxConnections 1
InitialR2T Yes
ImmediateData No
MaxRecvDataSegmentLength 131072
MaxXmitDataSegmentLength 131072
MaxBurstLength 262144
FirstBurstLength 262144
DefaultTime2Wait 2
DefaultTime2Retain 20
MaxOutstandingR2T 8
DataPDUInOrder Yes
DataSequenceInOrder Yes
ErrorRecoveryLevel 0
Lun 0 Path=/dev/san/racshare,Type=blockio,ScsiSN=4YMdbG-SGED-jqHA,ScsiId=4YMdbG-SGED-jqHA,IOMode=wt
[root@san etc]#
Make iSCSI Target(s) Available to Client(s)
Every time a new logical volume is added, you will need to restart the associated service on the Openfiler server. In my case, I created a new
iSCSI logical volume so I needed to restart the iSCSI target (iscsi-target) service. This will make the new iSCSI target available to all
clients on the network who have privileges to access it.
To restart the iSCSI target service, use the Openfiler Storage Control Center and navigate to [Services] / [Enable/Disable]. The iSCSI target
service should already be enabled (several sections back). If so, disable the service then enable it again. (See Figure 2)
The same task can be achieved through an SSH session on the Openfiler server:
[root@openfiler1 ~]# service iscsi-target restart
Stopping iSCSI target service: [ OK ]
Starting iSCSI target service: [ OK ]
Configure iSCSI Initiator and New Volume
An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a driver) is available. In this article, the
client is an Oracle database server (linux3) running CentOS 5.
In this section I will be configuring the iSCSI software initiator on the Oracle database server linux3. Red Hat Enterprise Linux (and CentOS 5)
includes the Open-iSCSI software initiator which can be found in the iscsi-initiator-utils RPM.
This is a change from previous versions of RHEL (4.x) which included the Linux iscsi-sfnet
software driver developed as part of the Linux-iSCSI Project.
All iSCSI management tasks like discovery and logins will use the command-line interface iscsiadm which is included with Open-iSCSI.
The iSCSI software initiator on linux3 will be configured to automatically login to the network storage server (openfiler1) and discover the
iSCSI volume created in the previous section. I will then go through the steps of creating a persistent local SCSI device name (i.e.
/dev/iscsi/linux3-data-1) for the iSCSI target name discovered using udev. Having a consistent local SCSI device name and which
iSCSI target it maps to is highly recommended in order to distinguish between multiple SCSI devices. Before I can do any of this, however, I
must first install the iSCSI initiator software!
Connecting to an iSCSI Target with Open-iSCSI Initiator using Linux
http://www.idevelopment.info/data/Unix/Linux/LINUX_ConnectingToAniSCSITargetWithOpen-iSCSIInitiatorUsingLinux.shtml[2015/3/31 9:11:34]
Installing the iSCSI (Initiator) Service
With Red Hat Enterprise Linux 5 (and CentOS 5), the Open-iSCSI iSCSI software initiator does not get installed by default. The software is
included in the iscsi-initiator-utils package which can be found on CD #1. To determine if this package is installed (which in most
cases, it will not be), perform the following on the client node (linux3):
[root@linux3 ~]# rpm -qa | grep iscsi-initiator-utils
If the iscsi-initiator-utils package is not installed, load CD #1 into the machine and perform the following:
[root@linux3 ~]# mount -r /dev/cdrom /media/cdrom
[root@linux3 ~]# cd /media/cdrom/CentOS
[root@linux3 ~]# rpm -Uvh iscsi-initiator-utils-6.2.0.865-0.8.el5.i386.rpm
[root@linux3 ~]# cd /
[root@linux3 ~]# eject
Configure the iSCSI (Initiator) Service
After verifying that the iscsi-initiator-utils package is installed, start the iscsid service and enable it to automatically start when the
system boots. I will also configure the iscsi service to automatically start which logs into iSCSI targets needed at system start up.
[root@linux3 ~]# service iscsid start
Turning off network shutdown. Starting iSCSI daemon: [ OK ]
[ OK ]
[root@linux3 ~]# chkconfig iscsid on
[root@linux3 ~]# chkconfig iscsi on
Now that the iSCSI service is started, use the iscsiadm command-line interface to discover all available targets on the network storage server:
[root@linux3 ~]# iscsiadm -m discovery -t sendtargets -p openfiler1
当发现不了iscsi target
修改/etc/initiators.deny注释掉所有的行
Manually Login to iSCSI Target(s)
At this point the iSCSI initiator service has been started and the client node was able to discover the available target(s) from the network storage
server. The next step is to manually login to the available target(s) which can be done using the iscsiadm command-line interface. Note that I
had to specify the IP address and not the host name of the network storage server (openfiler1-san) - I believe this is required given the
discovery (above) shows the targets using the IP address.
[root@linux3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:scsi.linux3-data-1 -p
192.168.2.195 --login
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1, portal:
192.168.2.195,3260]
Login to [iface: default, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1, portal:
192.168.2.195,3260]: successful
Configure Automatic Login
The next step is to make certain the client will automatically login to the target(s) listed above when the machine is booted (or the iSCSI initiator
service is started/restarted):
[root@linux3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:scsi.linux3-data-1 -p
192.168.2.195 --op update -n node.startup -v automatic
Create Persistent Local SCSI Device Names
In this section, I will go through the steps to create a persistent local SCSI device name (/dev/iscsi/linux3-data-1) which will be mapped
to the new iSCSI target name. This will be done using udev. Having a consistent local SCSI device name (for example /dev/mydisk1 or
/dev/mydisk2) is highly recommended in order to distinguish between multiple SCSI devices (/dev/sda or /dev/sdb) when the node is
booted or the iSCSI initiator service is started/restarted.
When the database server node boots and the iSCSI initiator service is started, it will automatically login to the target(s) configured in a random
fashion and map them to the next available local SCSI device name. For example, the target iqn.2006-01.com.openfiler:scsi.linux3-
data-1 may get mapped to /dev/sda when the node boots. I can actually determine the current mappings for all targets (if there were multiple
targets) by looking at the /dev/disk/by-path directory:
[root@linux3 ~]# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " "
$11}')
ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:scsi.linux3-data-1 -> ../../sda
Using the output from the above listing, we can establish the following current mappings:
Current iSCSI Target Name to local SCSI Device Name Mappings
Ok, so I only have one target discovered which maps to /dev/sda. But what if there were multiple targets configured (say, iqn.2006-
01.com.openfiler:scsi.linux3-data-2) or better yet, I had multiple removable SCSI devices on linux3? This mapping could change
every time the node is rebooted. For example, if I had a second target discovered on linux3 (i.e. iqn.2006-
01.com.openfiler:scsi.linux3-data-2), after a reboot it may be determined that the second iSCSI target iqn.2006-
01.com.openfiler:scsi.linux3-data-2 gets mapped to the local SCSI device /dev/sda and iqn.2006-
01.com.openfiler:scsi.linux3-data-1 gets mapped to the local SCSI device /dev/sdb or visa-versa.
As you can see, it is impractical to rely on using the local SCSI device names like /dev/sda or /dev/sdb given there is no way to predict the
iSCSI target mappings after a reboot.
What we need is a consistent device name we can reference like /dev/iscsi/linux3-data-1 that will always point to the appropriate iSCSI
target through reboots. This is where the Dynamic Device Management tool named udev comes in. udev provides a dynamic device directory
using symbolic links that point to the actual device using a configurable set of rules. When udev receives a device event (for example, the client
logging in to an iSCSI target), it matches its configured rules against the available device attributes provided in sysfs to identify the device. Rules
that match may provide additional device information or specify a device node name and multiple symlink names and instruct udev to run
additional programs (a SHELL script for example) as part of the device event handling process.
The first step is to create a new rules file. This file will be named /etc/udev/rules.d/55-openiscsi.rules and contain only a single line
of name=value pairs used to receive events we are interested in. It will also define a call-out SHELL script
(/etc/udev/scripts/iscsidev.sh) to handle the event.
Create the following rules file /etc/udev/rules.d/55-openiscsi.rules on the client node linux3:
# /etc/udev/rules.d/55-openiscsi.rules
KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh
%b",SYMLINK+="iscsi/%c/part%n"
Next, create the UNIX SHELL script that will be called when this event is received. Let's first create a separate directory on the linux3 node
where udev scripts can be stored:
[root@linux3 ~]# mkdir -p /etc/udev/scripts
Finally, create the UNIX shell script /etc/udev/scripts/iscsidev.sh:
#!/bin/sh
# FILE: /etc/udev/scripts/iscsidev.sh
BUS=${1}
HOST=${BUS%%:*}
[ -e /sys/class/iscsi_host ] || exit 1
file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname"
target_name=$(cat ${file})
# This is not an open-scsi drive
if [ -z "${target_name}" ]; then
exit 1
fi
# Check if QNAP drive
check_qnap_target_name=${target_name%%:*}
if [ $check_qnap_target_name = "iqn.2004-04.com.qnap" ]; then
target_name=`echo "${target_name%.*}"`
fi
echo "${target_name##*.}"
After creating the UNIX SHELL script, change it to executable:
[root@linux3 ~]# chmod 755 /etc/udev/scripts/iscsidev.sh
Now that udev is configured, restart the iSCSI initiator service
[root@linux3 ~]# service iscsi stop
Logging out of session [sid: 3, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1, portal:
192.168.2.195,3260]
Logout of [sid: 3, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1, portal:
192.168.2.195,3260]: successful
Stopping iSCSI daemon: /etc/init.d/iscsi: line 33: 5143 Killed
/etc/init.d/iscsid stop
[root@linux3 ~]# service iscsi start
iscsid dead but pid file exists
Turning off network shutdown. Starting iSCSI daemon: [ OK ]
[ OK ]
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-
01.com.openfiler:scsi.linux3-data-1, portal: 192.168.2.195,3260]
Login to [iface: default, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1, portal:
192.168.2.195,3260]: successful
[ OK ]
Let's see if our hard work paid off:
[root@linux3 ~]# ls -l /dev/iscsi/
total 0
drwxr-xr-x 2 root root 60 Apr 7 01:57 linux3-data-1
[root@linux3 ~]# ls -l /dev/iscsi/linux3-data-1/
total 0
lrwxrwxrwx 1 root root 9 Apr 7 01:57 part -> ../../sda
The listing above shows that udev did the job is was suppose to do! We now have a consistent set of local device name(s) that can be used to
reference the iSCSI targets through reboots. For example, we can safely assume that the device name /dev/iscsi/linux3-data-1/part
will always reference the iSCSI target iqn.2006-01.com.openfiler:scsi.linux3-data-1. We now have a consistent iSCSI target name to local device name mapping which is described in the following table:
Create Primary Partition on iSCSI Volume
I now need to create a single primary partition on the new iSCSI volume that spans the entire size of the volume. The fdisk command is used
in Linux for creating (and removing) partitions. You can use the default values when creating the primary partition as the default action is to use
the entire disk. You can safely ignore any warnings that may indicate the device does not contain a valid DOS partition (or Sun, SGI or OSF
disklabel).
[root@linux3 ~]# fdisk /dev/iscsi/linux3-data-1/part
Command (m for help): n
Command action
e extended
p p primary partition (1-4)
Partition number (1-4): 1
First cylinder (1-36864, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-36864, default 36864): 36864
Command (m for help): p
Disk /dev/iscsi/linux3-data-1/part: 38.6 GB, 38654705664 bytes
64 heads, 32 sectors/track, 36864 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/iscsi/linux3-data-1/part1 1 36864 37748720 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Create File System on new iSCSI Volume / Partition
The next step is to create an ext3 file system on the new partition. Provided with the RHEL distribution is a script named /sbin/mkfs.ext3
which makes the task of creating an ext3 file system seamless. Here is an example session of using the mkfs.ext3 script on linux3:
[root@linux3 ~]# mkfs.ext3 -b 4096 /dev/iscsi/linux3-data-1/part1
Connecting to an iSCSI Target with Open-iSCSI Initiator using Linux
http://www.idevelopment.info/data/Unix/Linux/LINUX_ConnectingToAniSCSITargetWithOpen-iSCSIInitiatorUsingLinux.shtml[2015/3/31 9:11:34]
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
4718592 inodes, 9437180 blocks
471859 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
288 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Mount the New File System
Now that the new iSCSI volume is partition and formatted, the final step is to mount the new volume. For this example, I will be mounting the
new volume on the directory /u03.
Create the /u03 directory before attempting to mount the new volume:
[root@linux3 ~]# mkdir -p /u03
Next, edit the /etc/fstab on linux3 and add an entry for the new volume:
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
//ddeevv//ViosclGsir/oluip0n0u/x3Lo-gdVaotla0-11 / p a r t1 s/wua0p3 sewxatp3 d_enfeatudletvs 00 00
cartman:SHARE2 /cartman nfs defaults 0 0
domo:Public /domo nfs defaults 0 0
After making the new entry in the /etc/fstab file, it is now just a matter of mounting the new iSCSI volume:
[root@linux3 ~]# mount /u03
[root@linux3 ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
56086828 21905480 31286296 42% /
/dev/hda1 101086 19160 76707 20% /boot
tmpfs 1037056 0 1037056 0% /dev/shm
cartman:SHARE2 306562280 8448 306247272 1% /cartman
/ddoemov/:Psduab1li c 1 93179175862490102 3 2 9158109274404 13559008286732146 8 11%8 %/ u/0d3omo
Logout and Remove an iSCSI Target from a Linux Client
It is my hope that this article has provided valuable insight into how you can take advantage of networked storage and the iSCSI configuration
process. As you can see, the process is fairly straightforward. Just as simple as it was to configure the Open-iSCSI Initiator on Linux, it is just as
easy to remove it and that is the subject of this section.
1. Unmount the File System
[root@linux3 ~]# cd
[root@linux3 ~]# umount /u03
After unmounting the file system, remove (or comment out) its related entry from the /etc/fstab file:
# /dev/iscsi/linux3-data-1/part1 /u03 ext3 _netdev 0 0
2. Manually Logout of iSCSI Target(s)
[root@linux3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:scsi.linux3-data-1 –p
192.168.2.195 --logout
Logging out of session [sid: 4, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1,
portal: 192.168.2.195,3260]
Logout of [sid: 4, target: iqn.2006-01.com.openfiler:scsi.linux3-data-1, portal:
192.168.2.195,3260]: successful
Verify we are logged out of the iSCSI target by looking at the /dev/disk/by-path directory. If no other iSCSI targets exist on the client
node, then after logging out from the iSCSI target, the mappings for all targets should be gone and the following command should not
find any files or directories:
[root@linux3 ~]# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10
" " $11}')
ls: *openfiler*: No such file or directory
3. Delete Target and Disable Automatic Login
Update the record entry on the client node to disable automatic logins to the iSCSI target:
[root@linux3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:scsi.linux3-data-1 -p
192.168.2.195 --op update -n node.startup -v manual
Delete the iSCSI target:
[root@linux3 ~]# iscsiadm -m node --op delete --targetname iqn.2006-
01.com.openfiler:scsi.linux3-data-1
4. Remove udev Rules Files
If the iSCSI target being removed is the only remaining target and you don't plan on adding any further iSCSI targets in the future, then it
is safe to remove the iSCSI rules file and its call-out script:
[root@linux3 ~]# rm /etc/udev/rules.d/55-openiscsi.rules
[root@linux3 ~]# rm /etc/udev/scripts/iscsidev.sh
5. Disable the iSCSI (Initiator) Service
If the iSCSI target being removed is the only remaining target and you don't plan on adding any further iSCSI targets in the future, then it
is safe to disable the iSCSI Initiator Service:
[root@linux3 ~]# service iscsid stop
[root@linux3 ~]# chkconfig iscsid off
[root@linux3 ~]# chkconfig iscsi off
About the Author
Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as
a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning,
Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database
design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics,
programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of
course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at:
http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer
Science and Mathematics.
