[20260421]11g关于hugepages的相关问题.txt
[20260421]11g关于hugepages的相关问题.txt
--//昨天测试21c设置use_large_pages时遇到的问题，遇到以前在11g测试不同的情况，在11g重复测试验证看看。
--//测试1.设置use_large_pages＝auto，是否启动oaadism命令。
--//测试2.设置use_large_pages＝only时，vm.nr_overcommit_hugepages是否有效。
--//测试3，无论设置use_large_pages任何值，是否都会执行oradism。
1.环境：
SYS@book> @ ver2
==============================
PORT_STRING : x86_64/Linux 2.4.xx
VERSION : 11.2.0.4.0
BANNER : Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL procedure successfully completed.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
# grep -i hugepage /proc/meminfo
AnonHugePages: 16384 kB
HugePages_Total: 385
HugePages_Free: 118
HugePages_Rsvd: 118
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//当前实例使用hugepages数量 HugePages_Total-HugePages_Free+HugePages_Rsvd = 385.
SYS@book> @ pvalid use_large_pages
Display valid values for multioption parameters matching "use_large_pages"...
PAR# PARAMETER ORD VALUE DEFAULT
------ ---------------- --- ------- -------
117 use_large_pages 1 TRUE DEFAULT
use_large_pages 2 AUTO
use_large_pages 3 ONLY
use_large_pages 4 FALSE
--//11g仅仅支持4种设置，不支持auto_only设置。
SYS@book> @ hidez use_large_pages
SYS@book> @ pr
==============================
NUM : 117
N_HEX : 75
NAME : use_large_pages
DESCRIPTION : Use large pages if available (TRUE/FALSE/ONLY)
DEFAULT_VALUE : FALSE
SESSION_VALUE : ONLY
SYSTEM_VALUE : ONLY
ISSES_MODIFIABLE : FALSE
ISSYS_MODIFIABLE : FALSE
PL/SQL procedure successfully completed.
--//当前use_large_pages=only。
2.测试1：
--//设置use_large_pages＝auto，是否启动oaadism命令。
SYS@book> alter system set use_large_pages=auto scope=spfile ;
System altered.
SYS@book> shutdown immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.
--//修改/etc/sysctl.d/98-oracle.conf，vm.nr_hugepages = 350，vm.nr_overcommit_hugepages = 50。
--//在关闭数据库的情况执行sysctl -p /etc/sysctl.d/98-oracle.conf.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 350
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
....
SYS@book> startup nomount
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//可以启动数据库。
--//查看alert日志：
Tue Apr 21 09:15:19 2026
Adjusting the default value of parameter parallel_max_servers
from 160 to 150 due to the value of parameter processes (180)
Starting ORACLE instance (normal)
DISM started, OS id=3422
~~~~~~~~~~~~~~~~~~~~~~~~~
************************ Large Pages Information *******************
Parameter use_large_pages = AUTO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 770 MB (100%)
Large Pages used by this instance: 385 (770 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 385 (770 MB)
Large Page size = 2048 KB
Time taken to allocate Large Pages = 0.002841 sec
********************************************************************
--//注意看下划线，启动dism。
$ ps -fp 3422
UID PID PPID C STIME TTY TIME CMD
root 3422 1 0 09:15 ? 00:00:00 ora_dism_book
# grep -i hugepage /proc/meminfo
AnonHugePages: 20480 kB
HugePages_Total: 385
HugePages_Free: 161
HugePages_Rsvd: 161
HugePages_Surp: 0
Hugepagesize: 2048 kB
# sysctl vm.nr_hugepages
vm.nr_hugepages = 385
--//修改内核参数满足启动需求。
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 294912 oracle 640 12582912 18
0x00000000 327681 oracle 640 792723456 18
0x56c108b0 360450 oracle 640 2097152 18
# ipcs -mu
------ Shared Memory Status --------
segments allocated 3
pages allocated 197120
pages resident 114688
pages swapped 0
Swap performance: 0 attempts 0 successes
--//12582912/2/1024/1024 = 6
--//792723456 /2/1024/1024 = 378
--//2097152 /2/1024/1024 = 1
--//6+378+1 = 385,正好满足启动需求。
# ps -fp 3422 -l
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 S root 3422 1 0 80 0 - 200696 pipe_w 09:15 ? 00:00:00 ora_dism_book
# ps -p 3422 -l
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 3422 1 0 80 0 - 200696 pipe_w ? 00:00:00 oradism
--//昏，仔细看笔记，如果这样执行ps -p 3422 -l看到cmd显示的是oradism，而不是ora_dism_book。
--//也就是后台执行的是oradism。
$ ll -i $(which oradism)
20511540 -rwsr-x---. 1 root oinstall 71790 2013-08-24 10:51:02 /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//注意看owner，以及文件权限-rwsr-x---。
# lsof -p 3422
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
oradism 3422 root cwd DIR 8,17 115 33943558 /u01/app/oracle/product/11.2.0.4/db_1/dbs
oradism 3422 root rtd DIR 8,3 235 64 /
oradism 3422 root txt REG 8,17 71790 20511540 /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
oradism 3422 root DEL REG 0,12 294912 /SYSV00000000
oradism 3422 root DEL REG 0,12 327681 /SYSV00000000
oradism 3422 root DEL REG 0,12 360450 /SYSV56c108b0
oradism 3422 root mem REG 8,3 2116736 145558 /usr/lib64/libc-2.17.so
oradism 3422 root mem REG 8,3 113152 145568 /usr/lib64/libnsl-2.17.so
oradism 3422 root mem REG 8,3 143352 145584 /usr/lib64/libpthread-2.17.so
oradism 3422 root mem REG 8,3 1141456 145566 /usr/lib64/libm-2.17.so
oradism 3422 root mem REG 8,3 19344 145564 /usr/lib64/libdl-2.17.so
oradism 3422 root mem REG 8,3 155064 145551 /usr/lib64/ld-2.17.so
oradism 3422 root 0r FIFO 0,8 0t0 24015 pipe
oradism 3422 root 1w FIFO 0,8 0t0 24016 pipe
oradism 3422 root 2r CHR 1,3 0t0 1028 /dev/null
oradism 3422 root 3r CHR 1,3 0t0 1028 /dev/null
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
oradism 3422 root txt REG 8,17 71790 20511540 /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//看来21c我的分析有问题，不过21c视乎每次启动oracle实例，不管use_large_pages如果设置，都会调用oradism。
--//但是21c下仅仅auto，auto_only，only才会在实例启动后，如果vm.nr_hugepages不足，后台运行的oradism保留在后台运行。
SYS@book> shutdown immediate ;
ORA-01507: database not mounted
ORACLE instance shut down.
# sysctl vm.nr_hugepages
vm.nr_hugepages = 385
# grep -i hugepage /proc/meminfo
AnonHugePages: 10240 kB
HugePages_Total: 385
HugePages_Free: 385
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//可以发现关闭数据库不会修改回来内核参数vm.nr_hugepages = 350。
3.测试2：
--//设置use_large_pages＝only时，vm.nr_overcommit_hugepages是否有效。
SYS@book> alter system set use_large_pages=only scope=spfile ;
System altered.
SYS@book> shutdown immediate ;
ORA-01507: database not mounted
ORACLE instance shut down.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 350
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
...
# sysctl vm.nr_hugepages
vm.nr_hugepages = 350
SYS@book> startup nomount
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//如果看21下的测试，这类情况无法启动实例。
--//alert日志如下：
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 770 MB (100%)
Large Pages used by this instance: 385 (770 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 385 (770 MB)
Large Page size = 2048 KB
********************************************************************
--//并没有启动dism。
# ps -ef | grep ora_dis[m]
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//后台不存在oradism的程序在执行。
# sysctl vm.nr_hugepages
vm.nr_hugepages = 350
--//可以发现内核vm.nr_hugepages没有修改。
# grep -i hugepage /proc/meminfo
AnonHugePages: 20480 kB
HugePages_Total: 385
HugePages_Free: 161
HugePages_Rsvd: 161
HugePages_Surp: 35
~~~~~~~~~~~~~~~~~~~~~~~~
Hugepagesize: 2048 kB
--//注意看下划线HugePages_Surp=35。等于像参数vm.nr_overcommit_hugepages借了35个hugepages。这样
--//350+35 = 385.
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 557056 oracle 640 12582912 17
0x00000000 589825 oracle 640 792723456 17
0x56c108b0 622594 oracle 640 2097152 17
# ipcs -mu
------ Shared Memory Status --------
segments allocated 3
pages allocated 197120
pages resident 114688
pages swapped 0
Swap performance: 0 attempts 0 successes
SYS@book> shutdown immediate ;
ORA-01507: database not mounted
ORACLE instance shut down.
# sysctl vm.nr_hugepages
vm.nr_hugepages = 350
# grep -i hugepage /proc/meminfo
AnonHugePages: 12288 kB
HugePages_Total: 350
HugePages_Free: 350
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//内核参数vm.nr_hugepages依旧是350.相当于收回35个hugepages。
4.测试3：
--//测试3，设置use_large_pages任何值，是否都会执行oradism。
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
--//use_large_pages=only.
SYS@book> alter system set use_large_pages=only scope=spfile ;
System altered.
--//关闭数据库略。
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//可以正常启动实例，说明在use_large_pages=only下没有启动dism。
--//注意几点，注意这样strace会产生许多aaa.<spid>的文件，建议测试前单独建立一个空目录，在此目录下执行strace命令。
$ ls -l aaa.*|wc
43 344 2837
--//另外要关闭strace的执行跟踪，需要打开另外会话关闭数据库命令。
--//后续的测试略，仅仅记录测试结果。
--//use_large_pages=false
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//use_large_pages=true
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//use_large_pages=auto
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//可以发现与测试1的情况完成不同。因为vm.nr_hugepages = 385，满足启动实例需求，不会启动dism。
--//use_large_pages=auto
--//修改vm.nr_hugepages = 350。
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 350
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
--//重复测试:
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//嗯,居然可以正常启动.
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 1605632 oracle 640 12582912 16
0x00000000 1638401 oracle 640 398458880 16
0x00000000 1671170 oracle 640 197132288 16
0x00000000 1703939 oracle 640 100663296 16
0x00000000 1736708 oracle 640 25165824 16
0x00000000 1769477 oracle 640 71303168 16
0x56c108b0 1802246 oracle 640 2097152 16
--//可以发现实际上使用混合模式.
# ipcs -mu
------ Shared Memory Status --------
segments allocated 7
pages allocated 197120
pages resident 112985
pages swapped 0
Swap performance: 0 attempts 0 successes
--//alert日志:
Starting ORACLE instance (normal)
WARNING: -------------------------------
WARNING: oradism did not start up correctly.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Return code: 16 errno 0 info1 54321 info2 65535
----------------------------------------
Oradism binary does not have root privilege.
Please verify if oradism has required privilege
Oradism Spawned failed for largepages allocation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
************************ Large Pages Information *******************
Parameter use_large_pages = AUTO
Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 700 MB (90%)
Large Pages used by this instance: 350 (700 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 350 (700 MB)
Large Page size = 2048 KB
Time taken to allocate Large Pages = 0.000000 sec
RECOMMENDATION:
Total System Global Area size is 770 MB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 35 (page size 2048 KB, total size 70 MB) system wide to
get 100% of the System Global Area allocated with large pages
********************************************************************
--//注意看下划线,oradism启动失败，实际上切换到use_large_pages = true的情况。
# grep -i hugepage /proc/meminfo
AnonHugePages: 26624 kB
HugePages_Total: 350
HugePages_Free: 161
HugePages_Rsvd: 161
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//过滤收集alert相关use_large_pages以及dism相关信息：
$ egrep -ni "Parameter use_large_pages = |dism" alert_book.log
5:DISM started, OS id=3422
7:Parameter use_large_pages = AUTO
128:Parameter use_large_pages = AUTO
251:Parameter use_large_pages = ONLY
371:Parameter use_large_pages = ONLY
491:Parameter use_large_pages = ONLY
613:Parameter use_large_pages = FALSE
732:Parameter use_large_pages = FALSE
973:Parameter use_large_pages = AUTO
1094:Parameter use_large_pages = AUTO
1215:WARNING: oradism did not start up correctly.
1218:Oradism binary does not have root privilege.
1219:Please verify if oradism has required privilege
1220:Oradism Spawned failed for largepages allocation
1222:Parameter use_large_pages = AUTO
--//可以仅仅在use_large_pages = AUTO的情况下，2次出现dism相关信息，结合整个测试过程，可以发现vm.nr_hugepages不足的情况下
--//才会启动dism,而21c的情况完全不同,不管use_large_pages设置什么，都会执行oradism命令，但是仅仅出现auto以及auto_only的情
--//况下保留后台进程ora_dism_<sid>。
5.收尾：
SYS@book> alter system set use_large_pages=only scope=spfile ;
System altered.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
--//昨天测试21c设置use_large_pages时遇到的问题，遇到以前在11g测试不同的情况，在11g重复测试验证看看。
--//测试1.设置use_large_pages＝auto，是否启动oaadism命令。
--//测试2.设置use_large_pages＝only时，vm.nr_overcommit_hugepages是否有效。
--//测试3，无论设置use_large_pages任何值，是否都会执行oradism。
1.环境：
SYS@book> @ ver2
==============================
PORT_STRING : x86_64/Linux 2.4.xx
VERSION : 11.2.0.4.0
BANNER : Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL procedure successfully completed.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
# grep -i hugepage /proc/meminfo
AnonHugePages: 16384 kB
HugePages_Total: 385
HugePages_Free: 118
HugePages_Rsvd: 118
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//当前实例使用hugepages数量 HugePages_Total-HugePages_Free+HugePages_Rsvd = 385.
SYS@book> @ pvalid use_large_pages
Display valid values for multioption parameters matching "use_large_pages"...
PAR# PARAMETER ORD VALUE DEFAULT
------ ---------------- --- ------- -------
117 use_large_pages 1 TRUE DEFAULT
use_large_pages 2 AUTO
use_large_pages 3 ONLY
use_large_pages 4 FALSE
--//11g仅仅支持4种设置，不支持auto_only设置。
SYS@book> @ hidez use_large_pages
SYS@book> @ pr
==============================
NUM : 117
N_HEX : 75
NAME : use_large_pages
DESCRIPTION : Use large pages if available (TRUE/FALSE/ONLY)
DEFAULT_VALUE : FALSE
SESSION_VALUE : ONLY
SYSTEM_VALUE : ONLY
ISSES_MODIFIABLE : FALSE
ISSYS_MODIFIABLE : FALSE
PL/SQL procedure successfully completed.
--//当前use_large_pages=only。
2.测试1：
--//设置use_large_pages＝auto，是否启动oaadism命令。
SYS@book> alter system set use_large_pages=auto scope=spfile ;
System altered.
SYS@book> shutdown immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.
--//修改/etc/sysctl.d/98-oracle.conf，vm.nr_hugepages = 350，vm.nr_overcommit_hugepages = 50。
--//在关闭数据库的情况执行sysctl -p /etc/sysctl.d/98-oracle.conf.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 350
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
....
SYS@book> startup nomount
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//可以启动数据库。
--//查看alert日志：
Tue Apr 21 09:15:19 2026
Adjusting the default value of parameter parallel_max_servers
from 160 to 150 due to the value of parameter processes (180)
Starting ORACLE instance (normal)
DISM started, OS id=3422
~~~~~~~~~~~~~~~~~~~~~~~~~
************************ Large Pages Information *******************
Parameter use_large_pages = AUTO
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 770 MB (100%)
Large Pages used by this instance: 385 (770 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 385 (770 MB)
Large Page size = 2048 KB
Time taken to allocate Large Pages = 0.002841 sec
********************************************************************
--//注意看下划线，启动dism。
$ ps -fp 3422
UID PID PPID C STIME TTY TIME CMD
root 3422 1 0 09:15 ? 00:00:00 ora_dism_book
# grep -i hugepage /proc/meminfo
AnonHugePages: 20480 kB
HugePages_Total: 385
HugePages_Free: 161
HugePages_Rsvd: 161
HugePages_Surp: 0
Hugepagesize: 2048 kB
# sysctl vm.nr_hugepages
vm.nr_hugepages = 385
--//修改内核参数满足启动需求。
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 294912 oracle 640 12582912 18
0x00000000 327681 oracle 640 792723456 18
0x56c108b0 360450 oracle 640 2097152 18
# ipcs -mu
------ Shared Memory Status --------
segments allocated 3
pages allocated 197120
pages resident 114688
pages swapped 0
Swap performance: 0 attempts 0 successes
--//12582912/2/1024/1024 = 6
--//792723456 /2/1024/1024 = 378
--//2097152 /2/1024/1024 = 1
--//6+378+1 = 385,正好满足启动需求。
# ps -fp 3422 -l
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 S root 3422 1 0 80 0 - 200696 pipe_w 09:15 ? 00:00:00 ora_dism_book
# ps -p 3422 -l
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 3422 1 0 80 0 - 200696 pipe_w ? 00:00:00 oradism
--//昏，仔细看笔记，如果这样执行ps -p 3422 -l看到cmd显示的是oradism，而不是ora_dism_book。
--//也就是后台执行的是oradism。
$ ll -i $(which oradism)
20511540 -rwsr-x---. 1 root oinstall 71790 2013-08-24 10:51:02 /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//注意看owner，以及文件权限-rwsr-x---。
# lsof -p 3422
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
oradism 3422 root cwd DIR 8,17 115 33943558 /u01/app/oracle/product/11.2.0.4/db_1/dbs
oradism 3422 root rtd DIR 8,3 235 64 /
oradism 3422 root txt REG 8,17 71790 20511540 /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
oradism 3422 root DEL REG 0,12 294912 /SYSV00000000
oradism 3422 root DEL REG 0,12 327681 /SYSV00000000
oradism 3422 root DEL REG 0,12 360450 /SYSV56c108b0
oradism 3422 root mem REG 8,3 2116736 145558 /usr/lib64/libc-2.17.so
oradism 3422 root mem REG 8,3 113152 145568 /usr/lib64/libnsl-2.17.so
oradism 3422 root mem REG 8,3 143352 145584 /usr/lib64/libpthread-2.17.so
oradism 3422 root mem REG 8,3 1141456 145566 /usr/lib64/libm-2.17.so
oradism 3422 root mem REG 8,3 19344 145564 /usr/lib64/libdl-2.17.so
oradism 3422 root mem REG 8,3 155064 145551 /usr/lib64/ld-2.17.so
oradism 3422 root 0r FIFO 0,8 0t0 24015 pipe
oradism 3422 root 1w FIFO 0,8 0t0 24016 pipe
oradism 3422 root 2r CHR 1,3 0t0 1028 /dev/null
oradism 3422 root 3r CHR 1,3 0t0 1028 /dev/null
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
oradism 3422 root txt REG 8,17 71790 20511540 /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//看来21c我的分析有问题，不过21c视乎每次启动oracle实例，不管use_large_pages如果设置，都会调用oradism。
--//但是21c下仅仅auto，auto_only，only才会在实例启动后，如果vm.nr_hugepages不足，后台运行的oradism保留在后台运行。
SYS@book> shutdown immediate ;
ORA-01507: database not mounted
ORACLE instance shut down.
# sysctl vm.nr_hugepages
vm.nr_hugepages = 385
# grep -i hugepage /proc/meminfo
AnonHugePages: 10240 kB
HugePages_Total: 385
HugePages_Free: 385
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//可以发现关闭数据库不会修改回来内核参数vm.nr_hugepages = 350。
3.测试2：
--//设置use_large_pages＝only时，vm.nr_overcommit_hugepages是否有效。
SYS@book> alter system set use_large_pages=only scope=spfile ;
System altered.
SYS@book> shutdown immediate ;
ORA-01507: database not mounted
ORACLE instance shut down.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 350
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
...
# sysctl vm.nr_hugepages
vm.nr_hugepages = 350
SYS@book> startup nomount
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//如果看21下的测试，这类情况无法启动实例。
--//alert日志如下：
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 770 MB (100%)
Large Pages used by this instance: 385 (770 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 385 (770 MB)
Large Page size = 2048 KB
********************************************************************
--//并没有启动dism。
# ps -ef | grep ora_dis[m]
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//后台不存在oradism的程序在执行。
# sysctl vm.nr_hugepages
vm.nr_hugepages = 350
--//可以发现内核vm.nr_hugepages没有修改。
# grep -i hugepage /proc/meminfo
AnonHugePages: 20480 kB
HugePages_Total: 385
HugePages_Free: 161
HugePages_Rsvd: 161
HugePages_Surp: 35
~~~~~~~~~~~~~~~~~~~~~~~~
Hugepagesize: 2048 kB
--//注意看下划线HugePages_Surp=35。等于像参数vm.nr_overcommit_hugepages借了35个hugepages。这样
--//350+35 = 385.
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 557056 oracle 640 12582912 17
0x00000000 589825 oracle 640 792723456 17
0x56c108b0 622594 oracle 640 2097152 17
# ipcs -mu
------ Shared Memory Status --------
segments allocated 3
pages allocated 197120
pages resident 114688
pages swapped 0
Swap performance: 0 attempts 0 successes
SYS@book> shutdown immediate ;
ORA-01507: database not mounted
ORACLE instance shut down.
# sysctl vm.nr_hugepages
vm.nr_hugepages = 350
# grep -i hugepage /proc/meminfo
AnonHugePages: 12288 kB
HugePages_Total: 350
HugePages_Free: 350
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//内核参数vm.nr_hugepages依旧是350.相当于收回35个hugepages。
4.测试3：
--//测试3，设置use_large_pages任何值，是否都会执行oradism。
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
--//use_large_pages=only.
SYS@book> alter system set use_large_pages=only scope=spfile ;
System altered.
--//关闭数据库略。
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//可以正常启动实例，说明在use_large_pages=only下没有启动dism。
--//注意几点，注意这样strace会产生许多aaa.<spid>的文件，建议测试前单独建立一个空目录，在此目录下执行strace命令。
$ ls -l aaa.*|wc
43 344 2837
--//另外要关闭strace的执行跟踪，需要打开另外会话关闭数据库命令。
--//后续的测试略，仅仅记录测试结果。
--//use_large_pages=false
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//use_large_pages=true
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//use_large_pages=auto
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
--//可以发现与测试1的情况完成不同。因为vm.nr_hugepages = 385，满足启动实例需求，不会启动dism。
--//use_large_pages=auto
--//修改vm.nr_hugepages = 350。
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 350
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
--//重复测试:
$ strace -ff -o aaa sqlplus -s -l / as sysdba <<<"startup nomount"
ORACLE instance started.
Total System Global Area 801701888 bytes
Fixed Size 2257520 bytes
Variable Size 285216144 bytes
Database Buffers 507510784 bytes
Redo Buffers 6717440 bytes
--//嗯,居然可以正常启动.
# lsof /u01/app/oracle/product/11.2.0.4/db_1/bin/oradism
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 1605632 oracle 640 12582912 16
0x00000000 1638401 oracle 640 398458880 16
0x00000000 1671170 oracle 640 197132288 16
0x00000000 1703939 oracle 640 100663296 16
0x00000000 1736708 oracle 640 25165824 16
0x00000000 1769477 oracle 640 71303168 16
0x56c108b0 1802246 oracle 640 2097152 16
--//可以发现实际上使用混合模式.
# ipcs -mu
------ Shared Memory Status --------
segments allocated 7
pages allocated 197120
pages resident 112985
pages swapped 0
Swap performance: 0 attempts 0 successes
--//alert日志:
Starting ORACLE instance (normal)
WARNING: -------------------------------
WARNING: oradism did not start up correctly.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Return code: 16 errno 0 info1 54321 info2 65535
----------------------------------------
Oradism binary does not have root privilege.
Please verify if oradism has required privilege
Oradism Spawned failed for largepages allocation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
************************ Large Pages Information *******************
Parameter use_large_pages = AUTO
Per process system memlock (soft) limit = 128 GB
Total Shared Global Region in Large Pages = 700 MB (90%)
Large Pages used by this instance: 350 (700 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 350 (700 MB)
Large Page size = 2048 KB
Time taken to allocate Large Pages = 0.000000 sec
RECOMMENDATION:
Total System Global Area size is 770 MB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 35 (page size 2048 KB, total size 70 MB) system wide to
get 100% of the System Global Area allocated with large pages
********************************************************************
--//注意看下划线,oradism启动失败，实际上切换到use_large_pages = true的情况。
# grep -i hugepage /proc/meminfo
AnonHugePages: 26624 kB
HugePages_Total: 350
HugePages_Free: 161
HugePages_Rsvd: 161
HugePages_Surp: 0
Hugepagesize: 2048 kB
--//过滤收集alert相关use_large_pages以及dism相关信息：
$ egrep -ni "Parameter use_large_pages = |dism" alert_book.log
5:DISM started, OS id=3422
7:Parameter use_large_pages = AUTO
128:Parameter use_large_pages = AUTO
251:Parameter use_large_pages = ONLY
371:Parameter use_large_pages = ONLY
491:Parameter use_large_pages = ONLY
613:Parameter use_large_pages = FALSE
732:Parameter use_large_pages = FALSE
973:Parameter use_large_pages = AUTO
1094:Parameter use_large_pages = AUTO
1215:WARNING: oradism did not start up correctly.
1218:Oradism binary does not have root privilege.
1219:Please verify if oradism has required privilege
1220:Oradism Spawned failed for largepages allocation
1222:Parameter use_large_pages = AUTO
--//可以仅仅在use_large_pages = AUTO的情况下，2次出现dism相关信息，结合整个测试过程，可以发现vm.nr_hugepages不足的情况下
--//才会启动dism,而21c的情况完全不同,不管use_large_pages设置什么，都会执行oradism命令，但是仅仅出现auto以及auto_only的情
--//况下保留后台进程ora_dism_<sid>。
5.收尾：
SYS@book> alter system set use_large_pages=only scope=spfile ;
System altered.
# grep -i ^vm /etc/sysctl.d/98-oracle.conf
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
# sysctl -p /etc/sysctl.d/98-oracle.conf
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.nr_hugepages = 385
vm.nr_overcommit_hugepages = 50
浙公网安备 33010602011771号