这是参考ORACLE FC+ASM+UDEV+FAILGROUP，自已搭建的基于ORACLE12.2.0.1.0，用两台主机加VM的5台虚拟机实现该个实验搭建。
Redhat extent rac
一.环境规划
主机IP，主机名
hostname public_ip priv-ip vip sid scan-ip
exrac1 192.168.0.70 192.168.4.30 192.168.0.71 extora1 192.168.0.74
exrac2 192.168.0.75 192.168.4.31 192.168.0.73 extora2 192.168.0.74
ASM磁盘组
DATA（10G) FRA(5G) REDO(2G) OCR(8G) GIMR(40G) T_OCR(8G)
FGA DATA_A FRA_A REDO_A OCR_A GIMR_A T_OCR
FGB DATA_B FRA_B REDO_B OCR_B GIMR_B
FGQ /OCR/VOTE_DISK
数据文件 归档文件 日志文件 OCR文件 资料管理库 临时OCR盘
存储规划
存储A （内网IP 192.168.4.40,外网问IP 192.168.0.40）
卷 容量 备注
OCR_A 8G
FRA_A 5G
REDO_A 2G
GIMR_A 40G
DATA_A 10G
T_OCR 8G
存储B（内网IP 192.168.4.41,外网问IP 192.168.0.42）
卷 容量 备注
OCR_B 8G
FRA_B 5G
REDO_B 2
GIMR_B 40G
DATA_B 10G
NFS
卷 容量 备注
/ocr/vote_disk 8g
二、存储部置
1.存储A
•BLOCK DEVICE MANAGERMENT
•VOLUME GROUP MANAGEMENT
T_OCR
OCR_A
GIMR-A
Manage Services
把 ISCSCI TARGET 点击modify boot 的ENABLE 把CURRENT STATUS置为RUNNING 也就是点击START
TARGET CONFIGURATION
这里存储A为iqn.2020-08.com.openfiler:tsn.extend1
LUNS MAPPING
2.存储B
•BLOCK Device Management
•VOLUME GROUP MANAGEMENT
gimr-b
data-b
fra-b
ocr-b
NETWORK ACCESS CONFIGURATION
这里的网段为 192.168.0.0，192.168.4.0
Manage Services
把 ISCSCI TARGET 点击modify boot 的ENABLE 把CURRENT STATUS置为RUNNING 也就是点击START
TARGET CONFIGURATION
这里存储B为iqn.2020-08.com.openfiler:tsn.extend2
三、在两台服务器连接二台存储
1.服务器A连接存储A
在iscsid.conf进行修改，把用户和密码设置好
vi /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
To set a CHAP username and password for initiator
authentication by the target(s), uncomment the following lines:
node.session.auth.username = root
node.session.auth.password = password123
保存退出后
重启一下 ISCISID 服务
systemctl restart iscsid.service
探测存储服务器
iscsiadm -m discovery -t sendtargets -p 192.168.4.40
挂载目标
iscsiadm -m node -T iqn.2020-08.com.openfiler:tsn.extent1 -p 192.168.4.40:3260 -l
然后
fdisk -l
2.服务器A连接存储B
探测存储服务器
iscsiadm -m discovery -t sendtargets -p 192.168.4.41
挂载目标
iscsiadm -m node -T iqn.2020-08.com.openfiler:tsn.extent2 -p 192.168.4.41：3260 -l
然后
fdisk -l
3.服务器B连接存储A
在iscsid.conf进行修改，把用户和密码设置好
vi /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
To set a CHAP username and password for initiator
authentication by the target(s), uncomment the following lines:
node.session.auth.username = root
node.session.auth.password = password123
保存退出后
重启一下 ISCISID 服务
systemctl restart iscsid.service
探测存储服务器
iscsiadm -m discovery -t sendtargets -p 192.168.4.40
挂载目标
iscsiadm -m node -T iqn.2020-08.com.openfiler:tsn.extend1 -p 192.168.4.40:3260 -l
4.服务器B连接存储B
探测存储服务器
iscsiadm -m discovery -t sendtargets -p 192.168.4.41
挂载目标
iscsiadm -m node -T iqn.2020-8.com.openfiler:tsn.extend2 -p 192.168.4.41:3260 -l
然后
fdisk -l
四 修改/etc/hosts
vi /etc/hosts
public ip
192.168.0.70 exrac1
192.168.0.75 exrac2
vip
192.168.0.71 exrac1-vip
192.168.0.73 exrac2-vip
private ip
192.168.4.40 exrac1-priv
192.168.4.41 exrac2-priv
scan ip
192.168.0.79 exrac-cluster-scan
五、内核参数文件
vi /etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
shmmax=（81024102410240.9)
kernel.shmmax = 7730941132
shmall=（210241024*0.9）
kernel.shmall = 1887436
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.tcp_rmem=4096 87380 4194304
net.ipv4.tcp_wmem=4096 16384 4194304
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.default.arp_filter = 2
net.ipv4.conf.all.rp_filter = 2
然后
sysctl -p
六、修改资源限制文件
vi /etc/security/limits.conf
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
七、关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
八、关闭 SELINUX
vi /etc/selinux/config
selinux=disabled
九 配置MTU参数（心跳网络将mtu调大至9000）：
vi /etc/sysconfig/network-scripts/ifcfg-ens39
MTU=9000
十 配置YUM源
mkdir /mnt/system
mount -o loop rhel-server-7.3-x86_64-dvd.iso /mnt/system
cd /etc/yum.repos.d/
vi rhel.repo
[rhel7]
name=rhel7
baseurl=file:///mnt/system
enabled=1
gpgcheck=0
修改共享内存分区大小
将如下参数添加到/etc/fstab，具体大小数值根据实际情况调整，因为这个数值和物理内存以及MEMORY_TARGET有关。
echo shm /dev/shm tmpfs size=12g 0 0 >> /etc/fstab
修改后，只需重新对shm进行挂载即可：
mount -o remount /dev/shm
十一通过YUM命令安装RPM
yum -y install bc binutils compat-libcap1 compat-libstdc++-33 compat-libstdc++-33.i686 e2fsprogs e2fsprogs-libs glibc glibc.i686 glibc-devel glibc-devel.i686 libaio libaio.i686 libaio-devel libaio-devel.i686 libX11 libX11.i686 libXau libXau.i686 libXi libXi.i686 libXtst libXtst.i686 libgcc libgcc.i686 libstdc++ libstdc++.i686 libstdc++-devel libstdc++-devel.i686 libxcb libxcb.i686 make smartmontools sysstat unzip lrzsz openssh-clients libXrender-devel.i686 libXrender-devel.x86_64 gcc gcc-c++ ksh elfutils-libelf-devel.x86_64
十二。创建用户
groupadd -g 500 oinstall
groupadd -g 501 dba
groupadd -g 502 oper
groupadd -g 503 asmadmin
groupadd -g 504 asmdba
groupadd -g 505 asmoper
useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
useradd -u 502 -g oinstall -G dba,asmdba,asmadmin oracle
passwd grid
passwd oracle
十三、创建相关目录
mkdir -p /u01/app/oracle
mkdir -p /u01/app/grid
mkdir -p /u01/app/grid_base
mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01
chown -R grid:oinstall /u01/app/grid
chown -R grid:oinstall /u01/app/grid_base
chown -R oracle:oinstall /u01/app/oracle
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01
启动NSCD服务
yum install -y nscd
systemctl enable nscd.service
systemctl start nscd.service
systemctl status nscd.service
十四、禁用NTP
systemctl stop ntpd.service
systemctl disable ntpd.service
mv /etc/ntp.conf /etc/ntp.conf.original
关闭LINUX透明大页
修改/etc/default/grub文件,如下是修改之后的效果：
[root@rac1 default]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet transparent_hugepage=never" ---->>> transparent_hugepage=never是新加的
GRUB_DISABLE_RECOVERY="true"
[root@rac1 default]# grub2-mkconfig -o /boot/grub2/grub.cfg ---->>执行该命令
关闭transparent_hugepage的配置
vi /etc/rc.local #注释：编辑rc.local文件，增加以下内容
if test -f /sys/kernel/mm/redhattransparenthugepage/enabled; then echo never > /sys/kernel/mm/redhattransparenthugepage/enabled
fi
修改NSSWITCH.CONF
vi /etc/nsswitch.conf
修改行
hosts: files dns
为
hosts: files dns nis
配置NOZEROCONF
vi /etc/sysconfig/network增加以下内容
NOZEROCONF=yes
检查 cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
十五 停止avahi-daemon服务
systemctl stop avahi-dnsconfd
systemctl stop avahi-daemon
systemctl disable avahi-dnsconfd
systemctl disable avahi-daemon
十六、配置用户环境变量i
1.GRID
vi /home/grid/.bash_profile
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid_base
export ORACLE_HOME=/u01/app/grid
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export PATH=$ORACLE_HOME/bin:$PATH
第二节点相应改成+ASM2
2.ORACLEor
export PATH
export ORACLE_SID=exora1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/db_home
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export PATH=$PATH:$ORACLE_HOME/bin
第二节点相应改成+exora2
十七 SSH互信配置
1.grid
[root@exrac1 ~]# su - grid
[grid@exrac1 ~]$ ssh-keygen -t rsa
[root@exrac2 ~]# su - grid
[grid@exrac2 ~]$ ssh-keygen -t rsa
[grid@exrac1 ~]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
[grid@exrac1 ~]$ ssh exrac2 cat /home/grid/.ssh/id_rsa.pub >> .ssh/authorized_keys
[grid@exrac1 ~]$ scp .ssh/authorized_keys exrac2:/home/grid/.ssh
配置完成后分别测试：
[grid@exrac1 ~]$ ssh exrac1 date
[grid@exrac1 ~]$ ssh exrac2 date
[grid@exrac1 ~]$ ssh exrac1-priv date
[grid@exrac1 ~]$ ssh exrac1-priv2 date
[grid@exrac1 ~]$ ssh exrac2-priv1 date
[grid@exrac1 ~]$ ssh exrac2-priv2 date
节点2同样重复执行，确认SSH互信没有问题
2.oracle
[root@exrac1 ~]# su - oracle
[oracle@exrac1 ~]$ ssh-keygen -t rsa
[oracle@exrac2 ~]# su - oracle
[oracle@exrac2 ~]$ ssh-keygen -t rsa
[oracle@exrac1 ~]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
[oracle@exrac1 ~]$ ssh exrac2 cat /home/orale/.ssh/id_rsa.pub >> .ssh/authorized_keys
[oracle@exrac1 ~]$ scp .ssh/authorized_keys exrac2:/home/oracle/.ssh
十八 多路径配置
安装多路径
rpm -qa|grep device-mapper
yum install device-mapper -y
启动多路径软件
modprobe dm-multipath
modprobe dm-round-robin
lsmod | grep multipath
设置开机启动
systemctl enable multipathd.service
systemctl start multipathd.service
生成配置文件
mpathconf --enable --find_multipaths y --with_module y --outfile /etc/multipath.conf
multipath -v0 加载
for i in c d e f g h i j k l m n o p;
do
echo "KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i", NAME="asm-disk$i", OWNER="grid", GROUP="asmadmin", MODE="0660""
done
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c455269734d7856772d466d79472d6e47394b", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c4552483046696e432d6e5350772d47527768", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45526a635065336d2d654f76762d41517564", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45526856784562792d304763722d586a4776", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c455269734d7856772d466d79472d6e47394b", NAME="asm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c4552483046696e432d6e5350772d47527768", NAME="asm-diskh", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45526856784562792d304763722d586a4776", NAME="asm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45527066467151652d6377705a2d744b6670", NAME="asm-diskj", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45527066467151652d6377705a2d744b6670", NAME="asm-diskk", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45524750494269302d6c4735332d7a744547", NAME="asm-diskl", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45524750494269302d6c4735332d7a744547", NAME="asm-diskm", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45526431566d78322d633965502d767a517a", NAME="asm-diskn", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c45526431566d78322d633965502d767a517a", NAME="asm-disko", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL"sd*", BUS"scsi", PROGRAM"/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT"14f504e46494c4552337256367a702d52436f692d784a4177", NAME="asm-diskp", OWNER="grid", GROUP="asmadmin", MODE="0660"
查询/etc/multipath/bindings
vi /etc/multipath/bindings
mpatha 14f504e46494c45526a635065336d2d654f76762d41517564
mpathb 14f504e46494c455269734d7856772d466d79472d6e47394b
mpathc 14f504e46494c4552483046696e432d6e5350772d47527768
mpathd 14f504e46494c45526856784562792d304763722d586a4776
mpathe 14f504e46494c45527066467151652d6377705a2d744b6670
mpathf 14f504e46494c45524750494269302d6c4735332d7a744547
mpathg 14f504e46494c45526431566d78322d633965502d767a517a
mpathh 14f504e46494c4552337256367a702d52436f692d784a4177
mpathi 14f504e46494c45525137537a67352d6b6633472d4343477a
根据BINDINGS文件添加到 WWIDs文件中
vi /etc/multipath/wwids
/14f504e46494c45526a635065336d2d654f76762d41517564/
/14f504e46494c455269734d7856772d466d79472d6e47394b/
/14f504e46494c4552483046696e432d6e5350772d47527768/
/14f504e46494c45526856784562792d304763722d586a4776/
/14f504e46494c45527066467151652d6377705a2d744b6670/
/14f504e46494c45524750494269302d6c4735332d7a744547/
/14f504e46494c45526431566d78322d633965502d767a517a/
/14f504e46494c4552337256367a702d52436f692d784a4177/
/14f504e46494c45525137537a67352d6b6633472d4343477a
配置多路径文件
blacklist {
devnode "^sda"
}
multipaths{
multipath{
wwid 14f504e46494c45527066467151652d6377705a2d744b6670
alias FRA_A
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c45526856784562792d304763722d586a4776
alias DATA_A
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c4552483046696e432d6e5350772d47527768
alias GIMR_A
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c455269734d7856772d466d79472d6e47394b
alias OCR_A
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c45526a635065336d2d654f76762d41517564
alias T_OCR
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c45525137537a67352d6b6633472d4343477a
alias OCR_B
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c4552337256367a702d52436f692d784a4177
alias FRA_B
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c45526431566d78322d633965502d767a517a
alias DATA_B
uid 0
mode 0600
}
multipath{
wwid 14f504e46494c45524750494269302d6c4735332d7a744547
alias GIRM_B
uid 0
mode 0600
}
}
重启多路径服务：
systemctl stop multipathd.service
systemctl start multipathd.service
systemctl sta8tus multipathd.service
查看多路径状态
multipath -ll
for i in ;
do
printf "%s %s\n" "$i" "$(udevadm info --query=all --name=/dev/mapper/$i | grep -i dm_uuid)";
done
十九 UDEV权限绑定
for i in b c d e f g h i j k
do
echo "KERNEL\“dm-*",SUBSYSTEM"block",PROGRAM"/sbin/iscsid --whitelisted --replace-whitespace --device=/dev/mapper/$name",RESULT"/sbin/iscsid --whitelisted --replace-whitespace --device=/dev/mapper/$name",SYMLINK+="asm-disk$i",OWNER="grid",GROUP="asmadmin",MODE="0660""
done
vi /etc/udev/rules.d/99-asm.rules
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c45526a635065336d2d654f76762d41517564
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c455269734d7856772d466d79472d6e47394b
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c4552483046696e432d6e5350772d47527768
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c45526856784562792d304763722d586a4776
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c45527066467151652d6377705a2d744b6670
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c45524750494269302d6c4735332d7a744547
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c45526431566d78322d633965502d767a517a
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c4552337256367a702d52436f692d784a4177
KERNEL"dm-*",OWNER="grid",GROUP="asmadmin",MODE="0660",ENV{DM_UUID}14f504e46494c45525137537a67352d6b6633472d4343477a
检查新的UDEV设备
/sbin/udevadm trigger --type=devices --action=change
/sbin/udevadm control --reload
二十 创建ASM磁盘
安装ASMLIB （PRM包i在/opt目录）：
rpm -ivh kmod-oracleasm-2.0.8-17.el7.x86_64.rpm
rpm -ivh oracleasm-support-2.1.8-3.el7.x86_64.rpm
rpm -ivh oracleasmlib-2.0.12-1.el7.x86_64.rpm
配置asmlib
/etc/init.d/oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
创建ASM DISK:
[root@exrac2 opt]# oracleasm deletedisk DATA_A
Clearing disk header: done
Dropping disk: done
[root@exrac2 opt]# oracleasm createdisk DATA_A /dev/mapper/DATA_A
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk DATA_B /dev/mapper/DATA_B
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk FRA_A /dev/mapper/FRA_A
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk FRA_B /dev/mapper/FRA_B
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk OCR_A /dev/mapper/OCR_A
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk OCR_B /dev/mapper/OCR_B
\Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk GIMR_A /dev/mapper/GIMR_A
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk GIMR_B /dev/mapper/GIMR_B
Writing disk header: done
Instantiating disk: done
[root@exrac2 opt]# oracleasm createdisk T_OCR /dev/mapper/T_OCR
Writing disk header: done
检查创建的ASM DISK
[root@exrac2 opt]# oracleasm listdisks
DATA_A
DATA_B
FRA_A
FRA_B
GIMR_A
GIMR_B
OCR_A
OCR_B
REDO_A
REDO_B
T_OCR
[root@exrac1 opt]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "DATA_A"
Instantiating disk "REDO_A"
Instantiating disk "GIMR_A"
Instantiating disk "T_OCR"
Instantiating disk "FRA_A"
Instantiating disk "OCR_A"
Instantiating disk "DATA_B"
Instantiating disk "REDO_B"
Instantiating disk "GIMR_B"
Instantiating disk "FRA_B"
Instantiating disk "OCR_B"
[root@exrac1 opt]# oracleasm listdisks
DATA_A
DATA_B
FRA_A
FRA_B
GIMR_A
GIMR_B
OCR_A
OCR_B
REDO_A
REDO_B
T_OCR
二十一、挂载NFS共享盘
首先在NFS服务器设置NFS共享目录(这里在192.168.0.43)服务器上
vi /etc/exports
/ocr 192.168.0.0/24(rw,sync,root_squash)
showmount -e 192.168.0.43
Export list for 192.168.0.43:
有两台服务器上对NFS盘MOUNT上
mkdir /ocr
mount -t nfs4 192.168.0.43:/ocr /ocr
vi /etc/fstab
192.168.0.43:/ocr /ocr nfs4 rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,ver
s=3,timeo=600 0 0
二十二 开启linux大页
计算需要的页数
linux一个大页的大小为2M，开启大页的总内存应该比sgamaxsize稍稍大一点，比如
sgamaxsize=180g，则： hugepages > (1801024)/2 = 92160
\配置**sysctl.conf文件，添加：***
vm.nr_hugepages = 92200
\配置/etc/security/limit.conf，添加****（比sgamaxsize稍大，以K为单位）：
oracle soft memlock 188800000
oracle hard memlock 188800000
\禁用透明大页（注意添加位置）😗***
添加 transparent_hugepage=never
vi /etc/grub.conf
Once modified the line will read similar to the following example:
title Oracle Linux Server (2.6.32-300.25.1.el6uek.x8664)
root (hd0,0)
kernel /vmlinuz-2.6.32-300.25.1.el6uek.x8664 ro root=LABEL=/
transparenthugepage=never
initrd /initramfs-2.6.32-300.25.1.el6uek.x8664.img
重启操作系统，验证大页是否被使用以及透明大页是否关闭：
cat /proc/meminfo | grep -i Huge
AnonHugePages为0，则表示已经关闭透明大页
HugePages_Total:1030
HugePages_Free: 774
HugePages_Rsvd: 769
表示已经开启大页
二十三 GRID软件安装
解压：unzip GI12.2.0.1.0.zip
两台机安装cvuqdisk-1.0.10-1.rpm
export CVUQDISK_GRP=oinstall
/u01/app/grid/cv/rpm
rpm -ivh cvuqdisk-1.0.10-1.rpm
检查配置环境
/u01/app/grid/runcluvfy.sh stage -pre crsinst -n exrac1,exrac2 -fixup -verbose
对GRID进行安装
虽注意，由于本版本为REDHAT7.3 所以安装时打补丁，不然ROOT.SH会出错
[grid@exrac1 grid]$ export DISPLAY=192.168.0.123:0.0
[grid@exrac1 grid]$ ./gridSetup.sh -applyOneOffs /opt/29301687/29301676/
设置暂时OCR
先点击Specify Failure Groups，配置FGA,FGB两个故障组
这里密码为gridpass
在root用户下依次运行 ORAINSTROOT.sh和root.sh
检查一下集群是否正常
[grid@exrac1 ~]$ crs_stat -t
Name Type Target State Host
ora....SM.lsnr ora....er.type ONLINE ONLINE exrac1
ora.GIMR.dg ora....up.type ONLINE ONLINE exrac1
ora....ER.lsnr ora....er.type ONLINE ONLINE exrac1
ora....AF.lsnr ora....er.type OFFLINE OFFLINE
ora....N1.lsnr ora....er.type ONLINE ONLINE exrac1
ora.MGMTLSNR ora....nr.type ONLINE ONLINE exrac1
ora.OCR.dg ora....up.type ONLINE ONLINE exrac1
ora.asm ora.asm.type ONLINE ONLINE exrac1
ora.chad ora.chad.type ONLINE ONLINE exrac1
ora.cvu ora.cvu.type ONLINE ONLINE exrac1
ora....C1.lsnr application ONLINE ONLINE exrac1
ora.exrac1.ons application ONLINE ONLINE exrac1
ora.exrac1.vip ora....t1.type ONLINE ONLINE exrac1
ora....C2.lsnr application ONLINE ONLINE exrac2
ora.exrac2.ons application ONLINE ONLINE exrac2
ora.exrac2.vip ora....t1.type ONLINE ONLINE exrac2
ora.mgmtdb ora....db.type ONLINE ONLINE exrac1
ora....network ora....rk.type ONLINE ONLINE exrac1
ora.ons ora.ons.type ONLINE ONLINE exrac1
ora.proxy_advm ora....vm.type OFFLINE OFFLINE
ora.qosmserver ora....er.type ONLINE ONLINE exrac1
ora.scan1.vip ora....ip.type ONLINE ONLINE exrac1
配置ASM磁盘组
这里用SQL语句进行创建
grid@exrac1 ~]$ sqlplus / as sysasm
SQL> create diskgroup DATA NORMAL REDUNDANCY
2 FAILGROUP FG_A DISK '/dev/mapper/DATA_A'
3 FAILGROUP FG_B DISK '/dev/mapper/DATA_B'
4 ATTRIBUTE 'au_size'='1M',
5 'compatible.asm'='12.2',
6 'compatible.rdbms'='12.2';
SQL> create diskgroup FRA NORMAL REDUNDANCY
2 FAILGROUP FG_A DISK '/dev/mapper/FRA_A'
3 FAILGROUP FG_B DISK '/dev/mapper/FRA_B'
4 ATTRIBUTE 'au_size'='1M',
5 'compatible.asm'='12.2',
6 'compatible.rdbms'='12.2';
SQL> CREATE DISKGROUP OCR2 NORMAL REDUNDANCY
2 FAILGROUP FG_A DISK '/dev/mapper/OCR_A'
3 FAILGROUP FG_B DISK '/dev/mapper/OCR_B'
4 ATTRIBUTE 'au_size'='1M',
5 'compatible.asm'='12.2',
6 'compatible.rdbms'='12.2';
安装ORACLE软件
[oracle@exrac1 database]$ export DISPLAY=192.168.0.121:0.0
[oracle@exrac1 database]$ ./runInstaller
Starting Oracle Universal Installer...
执行ROOT.SH
[root@exrac1 ~]# /u01/app/oracle/product/db_home/root.sh Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/db_home
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@exrac2 ~]# /u01/app/oracle/product/db_home/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/db_home
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
进行DBCA建库
[oracle@exrac1 database]$ dbca
这里密码为oraclepass
\八、数据库参数调整
\重要参数调整：
\DB：
alter system set db_files = 2000 scope=spfile sid='';
alter system set "gcundo_affinity"=false scope=spfile sid='';
alter system set "gcpolicy_time"=0 scope=spfile sid='';
alter system set parallelmaxservers=32 scope=spfile sid='';
alter system set sessioncachedcursors=500 scope=spfile sid='';
alter system set open_cursors=500 scope=spfile sid='';
alter system set maxdumpfile_size='4096M' scope=spfile sid='';
alter system set "optimizeruse_feedback" = false scope=spfile sid='';
alter system set "cursorobsolete_threshold" = 100 scope=spfile sid='';
alter system set "optimizermax_permutations" = 200 scope=spfile sid='';
alter system set "serialdirect_read" = NEVER scope=both sid='';
alter system set audit_trail = NONE scope=spfile sid='';
alter system set paralleladaptivemulti_user = FALSE scope=spfile sid='';
alter system set parallelforceLocal = TRUE scope=spfile sid='';
alter system set undo_retention = 1800 scope=spfile sid='';
alter system set dbrecoveryfiledestsize=1000g scope=both sid='';
\ASM：****
alter system set "asmhbeatiowait"=120 scope=spfile sid='';
alter system set "datafilewriteerrorscrash_instance" = false scope=spfile ;
alter system set asm_diskstring='/dev/oracleasm/disks/','/ocr/' scope=both sid='';
alter system set largepoolsize=128m scope=spfile sid='';
\配置ASM磁盘组**属性（命令方式创建的已经配置过，无需执行，图形界面配置的需执行）：***
alter diskgroup OCR set attribute 'compatible.rdbms' = '12.2';
alter diskgroup DATA set attribute 'compatible.rdbms' = '12.2';
alter diskgroup FRA set attribute 'compatible.rdbms' = '12.2';
alter diskgroup REDO set attribute 'compatible.rdbms' = '12.2';
alter diskgroup GIMR set attribute 'compatible.rdbms' = '12.2';
\九、数据库配置
1， \创建表空间****
2， \创建用户并授权
\附：常用SQL命令
\磁盘组添加新的磁盘：**
alter diskgroup DATA add
FAILGROUP FGA disk '/dev/oracleasm/disks/DATA_A2'
FAILGROUP FGB disk '/dev/oracleasm/disks/DATA_B2'
rebalance power 11;
\迁移**votedisk和ocr:****
1,迁移votdisk
crsctl replace votedisk +OCR
crsctl query css votedisk
2，迁移ocr
ocrconfig -add +OCR
ocrconfig -delete +TOCR
ocrcheck
3，grid用户迁移spfile
asmcmd spget
asmcmd spmove '+TOCR/hkrac-cluster/ASMPARAMETERFILE/registry.253.826898607' '+OCR/hkrac-cluster/asmparameterfile/spfileasm.ora'
4，grid用户迁移PWDFILE
asmcmd pwmove --asm -f +TOCR/orapwasm +OCR/orapwasm
srvctl config asm
\删除unknown状态的磁盘：****
alter diskgroup DATA drop disk DATA_0001 FORCE;
\添加新的磁盘：
alter diskgroup OCR add failgroup FGB disk '/dev/oracleasm/disks/OCR_B' force rebalance power 11;
alter diskgroup DATA add failgroup FGB disk '/dev/oracleasm/disks/DATA_B' force rebalance power 11;
alter diskgroup FRA add failgroup FGB disk '/dev/oracleasm/disks/DATA_B' force rebalance power 11;
alter diskgroup REDO add failgroup FGB disk '/dev/oracleasm/disks/FRA_B' force rebalance power 11;
alter diskgroup GIMR add failgroup FGB disk '/dev/oracleasm/disks/FRA_B' force rebalance power 11;
\删除表决盘：**
alter diskgroup OCR drop QUORUM disk OCR_0002 FORCE;
alter diskgroup OCR drop disk OCR_0000 FORCE;
\添加新的表决盘磁盘**
alter diskgroup OCR add QUORUM FAILGROUP QFOCR disk '/ocr/vote_disk' rebalance power 11;
\设置ASM磁盘组属性：****
可以在创建磁盘组的时候配置
alter diskgroup OCR set attribute 'compatible.rdbms' = '12.2';
alter diskgroup DATA set attribute 'compatible.rdbms' = '12.2';
alter diskgroup FRA set attribute 'compatible.rdbms' = '12.2';
alter diskgroup REDO set attribute 'compatible.rdbms' = '12.2';
alter diskgroup GIMR set attribute 'compatible.rdbms' = '12.2';
\恢复offline的磁盘：****
alter diskgroup DATA online disks in failgroup FGA;
alter diskgroup FRA online disks in failgroup FGA;
alter diskgroup OCR online disks in failgroup FGA;
alter diskgroup REDO online disks in failgroup FGA;
alter diskgroup GIMR online disks in failgroup FGA;
\ASM常用视图**：****
select * from v$asm_disk;
select * from v$asm_diskgroup;
select * from v$asm_operation;
v$asm_alias
v$asm_attribute
v$asm_client
v$asmdiskiostat
v$asmdiskstat
v$asmdiskgroupstat
v$asm_file
v$asm_template
浙公网安备 33010602011771号