lightdb/postgresql中的MemoryContext out of memory原因分析及解决思路
内存上下文的设计思路可以参考src/backend/utils/mmgr/README。
https://www.pgcon.org/2019/schedule/attachments/514_introduction-memory-contexts.pdf
http://www.light-pg.com/docs/lightdb/13.3-22.2/lt_cheat_funcs.html
可以通过lt_cheat_funcs（该扩展对运行时无明显的侵入和干扰，可以在生产使用）扩展实时查询，如下：
zjh@postgres=# create extension lt_cheat_funcs ;
CREATE EXTENSION
zjh@postgres=# select * from pg_stat_get_memory_context();
name | parent | level | total_bytes | total_nblocks | free_bytes | free_chunks | used_bytes
--------------------------+--------------------+-------+-------------+---------------+------------+-------------+------------
TopMemoryContext | | 0 | 510640 | 12 | 23720 | 19 | 486920
dynahash | TopMemoryContext | 1 | 8192 | 1 | 1448 | 0 | 6744
不仅可以查询当前实例的MemoryContext树，还可以在每个语句执行结束前打印内存上下文。
shared_preload_libraries='lt_stat_statements,lt_stat_activity,lt_cheat_funcs,lt_prewarm,lt_cron,ltaudit,lt_hint_plan,lt_show_plans,lt_standby_forward,lt_pathman' lt_cheat_funcs.log_memory_context=on
2022-09-06 09:32:13.807665T ltsql zjh@postgres [local] client backend SELECT 00000[2022-09-06 09:28:47 CST] 0 [84905] STATEMENT: select 1;
TopMemoryContext: 494256 total in 11 blocks; 9760 free (11 chunks); 484496 used
dynahash: 8192 total in 1 blocks; 1448 free (0 chunks); 6744 used
TopTransactionContext: 8192 total in 1 blocks; 7784 free (2 chunks); 408 used
RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used
MessageContext: 32768 total in 3 blocks; 21504 free (6 chunks); 11264 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
dynahash: 16384 total in 2 blocks; 4592 free (2 chunks); 11792 used
TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TopPortalContext: 8192 total in 1 blocks; 7656 free (0 chunks); 536 used
PortalContext: 1024 total in 1 blocks; 592 free (0 chunks); 432 used
dynahash: 16384 total in 2 blocks; 3504 free (2 chunks); 12880 used
CacheMemoryContext: 524288 total in 7 blocks; 104920 free (0 chunks); 419368 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 496 free (1 chunks); 1552 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 776 free (1 chunks); 2296 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
dynahash: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
MdSmgr: 8192 total in 1 blocks; 7760 free (0 chunks); 432 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
dynahash: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
dynahash: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
ErrorContext: 8192 total in 1 blocks; 7928 free (7 chunks); 264 used
lt_cheat_funcs还有一个参数lt_cheat_funcs.exit_on_segv，控制segment fault的时候是否触发master进程宕机。因为这种概率发生极小，所以默认可以不用开启。一旦有异常某个参数值导致了越界，可以考虑临时补救。
lt_cheat_funcs对性能的影响。基于ltbench测试可知，其对性能的影响几乎可以忽略不计。
https://jnidzwetzki.github.io/2022/05/28/postgres-memory-context.html
共享内存的使用姿势 https://pgsql-hackers.postgresql.narkive.com/6VuZjbMh/shared-memory-and-memory-context-question
主要在两个c文件中：src/backend/utils/mmgr/aset.c是实现，src/backend/utils/mmgr/mcxt.c对外接口。如果想要干预或监控memorycontext，可以通过注册回调hook进行。
要想掌握memorycontext，还得理解发生out of memory的情况以及如何解决，如下：
client backend BIND 53200[2022-10-12 11:09:13 CST] 14856195 [402043] ERROR: out of memory client backend BIND 53200[2022-10-12 11:09:13 CST] 14856195 [402043] DETAIL: Failed on request of size 48 in memory context "MessageContext". client backend BIND 53200[2022-10-12 11:09:13 CST] 14856195 [402043] STATEMENT: insert into tbfundprftflow4 (ta_client,last_asset
client backend PARSE 53200[2022-10-12 11:09:55 CST] 0 [402230] ERROR: out of memory client backend PARSE 53200[2022-10-12 11:09:55 CST] 0 [402230] DETAIL: Failed on request of size 40 in memory context "CachedPlanQuery". client backend PARSE 53200[2022-10-12 11:09:55 CST] 0 [402230] STATEMENT: insert into tbfundprftflow15 (ta_client,last_asset,curr_income_ratio
client backend PARSE 53200[2022-10-12 11:09:13 CST] 0 [402043] ERROR: out of memory client backend PARSE 53200[2022-10-12 11:09:13 CST] 0 [402043] DETAIL: Failed on request of size 65536 in memory context "ErrorContext".
从源码可以看出，如果allocset中没有空闲chunk或申请的内存特别大（超过AllocSetContext.allocChunkLimit）时，都会调用c malloc申请内存，如下：
/* * AllocSetAlloc * Returns pointer to allocated memory of given size or NULL if * request could not be completed; memory is added to the set. * * No request may exceed: * MAXALIGN_DOWN(SIZE_MAX) - ALLOC_BLOCKHDRSZ - ALLOC_CHUNKHDRSZ * All callers use a much-lower limit. * * Note: when using valgrind, it doesn't matter how the returned allocation * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will * return space that is marked NOACCESS - AllocSetRealloc has to beware! */ static void * AllocSetAlloc(MemoryContext context, Size size) { AllocSet set = (AllocSet) context; AllocBlock block; AllocChunk chunk;
/* * Time to create a new regular (multi-chunk) block? */ if (block == NULL) { Size required_size; /* * The first such block has size initBlockSize, and we double the * space in each succeeding block, but not more than maxBlockSize. */ blksize = set->nextBlockSize; set->nextBlockSize <<= 1; if (set->nextBlockSize > set->maxBlockSize) set->nextBlockSize = set->maxBlockSize; /* * If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more * space... but try to keep it a power of 2. */ required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ; while (blksize < required_size) blksize <<= 1; /* Try to allocate it */ block = (AllocBlock) malloc(blksize);
至于错误日志中的“Failed on request of size 40 in memory context "CachedPlanQuery"”，是调用者打印的。在CachedPlanQuery内存上下文中找不到空闲可满足40字节的chunk，导致在allocsetalloc中新创建max(blksize = set->nextBlockSize,required_size)大的块（它是8KB,16KB,翻倍这么上去放在内存内存池，所以至少是申请8K失败，具体多少不确定，甚至有可能n GB，lightdb 23.1开始将会打印具体的值），此时通过malloc分配。如果malloc失败，就会直接返回NULL，没有打印实际申请多大的内存失败。从memorycontext本身申请是永远不会失败的。所以就要分析malloc失败的原因。
malloc()函数分配内存失败的常见原因：
1. 内存不足。
2. 在前面的程序中出现了内存的越界访问，导致malloc()分配函数所涉及的一些信息被破坏。下次再使用malloc()函数申请内存就会失败，返回空指针NULL(0)。
3. 没有连续内存能够满足申请。http://t.zoukankan.com/huty-p-8518846.html
4. 根据vm.overcommit_ratio和vm.overcommit_memory分析每个进程能够使用的内存大小。可以通过sar -r查看Committed_AS，通过/proc/meminfo查看Committed_AS、CommitLimit。如下：
CommitLimit = SwapTotal + MemTotal * overcommit_ratio
Committed_AS代表了系统已经分配的内存情况。
https://www.postgresql.org/message-id/4057e37d0fad0814281017dc6c211c00.squirrel@sq.gransy.com
