[20191202]关于hugepages相关问题.txt

原创 Oracle 作者：lfree 时间：2019-12-02 16:04:15 0 删除编辑

[20191202]关于hugepages相关问题.txt

--//上午在测试hugepages相关问题时遇到一个问题,自己以前疏忽了,就是在use_large_pages=true的情况下.
--//如果在/etc/sysctl.conf参数文件中vm.nr_hugepages定义不够大,即使定义vm.nr_overcommit_hugepages足够.
--//也不会向vm.nr_overcommit_hugepages借用内存,导致进入混合模式.
--//通过例子说明:

1.环境:
SCOTT@book> @ ver1
PORT_STRING                    VERSION        BANNER
------------------------------ -------------- --------------------------------------------------------------------------------
x86_64/Linux 2.4.xx            11.2.0.4.0     Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

SCOTT@book> alter system set use_large_pages=true scope=spfile;
System altered.

# grep "^vm.nr" /etc/sysctl.conf
vm.nr_hugepages = 104
vm.nr_overcommit_hugepages = 1000

# sysctl -p

2.测试:
--//重启数据库看看,检查alert文件发现:
Adjusting the default value of parameter parallel_max_servers
from 480 to 170 due to the value of parameter processes (200)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 51 GB

Total Shared Global Region in Large Pages = 208 MB (33%)

Large Pages used by this instance: 104 (208 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 104 (208 MB)
Large Page size = 2048 KB

RECOMMENDATION:
Total System Global Area size is 618 MB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 205 (page size 2048 KB, total size 410 MB) system wide to
get 100% of the System Global Area allocated with large pages
********************************************************************
--//可以发现仅仅使用定义vm.nr_hugepages = 104.

$ grep -i page /proc/meminfo
AnonPages:        165700 kB
PageTables:        18904 kB
AnonHugePages:         0 kB
HugePages_Total:     104
HugePages_Free:       49
HugePages_Rsvd:       49
HugePages_Surp:        0
Hugepagesize:       2048 kB

$ ps -ef | grep pmo[n] | awk '{print $2}' | xargs -IQ grep SYSV /proc/Q/numa_maps
60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=25 N0=1
60c00000 default file=/SYSV00000000\040(deleted) huge
6a400000 default file=/SYSV00000000\040(deleted) huge
6c400000 default file=/SYSV00000000\040(deleted) huge
6cc00000 default file=/SYSV00000000\040(deleted) huge
--//仅仅5个共享内存段使用hugepages.
6d000000 interleave:0-1 file=/SYSV00000000\040(deleted) dirty=785 mapmax=25 active=245 N0=400 N1=385
86800000 interleave:0-1 file=/SYSVe8a8ec10\040(deleted) dirty=1 mapmax=25 N1=1

$ sysresv
IPC Resources for ORACLE_SID "book" :
Shared Memory:
ID              KEY
413040641       0x00000000
413073410       0x00000000
413106179       0x00000000
413138949       0x00000000
413171718       0x00000000
413204487       0x00000000
413237256       0xe8a8ec10
Semaphores:
ID              KEY
335151104       0x6aa88594
Oracle Instance alive for sid "book"
--//也就是在use_large_pages=true的情况下不会向vm.nr_overcommit_hugepages参数借用内存.

3.测试:
SYS@book> alter system set use_large_pages=only scope=spfile;
System altered.

# grep "^vm.nr" /etc/sysctl.conf
vm.nr_hugepages = 104
vm.nr_overcommit_hugepages = 204

# sysctl -p
..

SYS@book> startup
ORA-27137: unable to allocate large pages to create a shared memory segment
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2097152

--//104+204 = 308,还不够大.
# grep "^vm.nr" /etc/sysctl.conf
vm.nr_hugepages = 104
vm.nr_overcommit_hugepages = 1000

# sysctl -p
..

SYS@book> startup
ORACLE instance started.
Total System Global Area 643084288 bytes
Fixed Size                  2255872 bytes
Variable Size             205521920 bytes
Database Buffers          427819008 bytes
Redo Buffers                7487488 bytes
Database mounted.
Database opened.

--//alert输出如下:
Adjusting the default value of parameter parallel_max_servers
from 480 to 170 due to the value of parameter processes (200)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
Per process system memlock (soft) limit = 51 GB

Large Pages unused system wide = 308 (616 MB)
Large Pages configured system wide = 308 (616 MB)
Large Page size = 2048 KB

ERROR:
Failed to allocate shared global region with large pages, unix errno = 12.
Aborting Instance startup.
ORA-27137: unable to allocate Large Pages to create a shared memory segment

ACTION:
Total System Global Area size is 624 MB. Prior to next instance restart:
1. Increase the number of unused large pages (page size 2048 KB)
to at least 312 (624 MB) to allocate 100% System Global Area
with large pages.
********************************************************************
--//以上是报ORA-27137的情况.

Mon Dec 02 15:59:56 2019
Adjusting the default value of parameter parallel_max_servers
from 480 to 170 due to the value of parameter processes (200)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
Per process system memlock (soft) limit = 51 GB

Total Shared Global Region in Large Pages = 618 MB (100%)

Large Pages used by this instance: 309 (618 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 309 (618 MB)
Large Page size = 2048 KB
********************************************************************

$ ps -ef | grep pmo[n] | awk '{print $2}' | xargs -IQ grep SYSV /proc/Q/numa_maps
60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=25 N0=1
60c00000 default file=/SYSV00000000\040(deleted) huge dirty=37 mapmax=25 N0=30 N1=7
86800000 interleave:0-1 file=/SYSVe8a8ec10\040(deleted) huge dirty=1 mapmax=25 N0=1

$ sysresv

IPC Resources for ORACLE_SID "book" :
Shared Memory:
ID              KEY
413401089       0x00000000
413433858       0x00000000
413466627       0xe8a8ec10
Semaphores:
ID              KEY
335314944       0x6aa88594
Oracle Instance alive for sid "book"

3.总结:
--//设置内核参数vm.nr_overcommit_hugepages ,仅仅在use_large_pages=only的情况下才会有效.