main/patch: excessive memory allocation
It seems that patch tries to allocate ~5GB of memory during patch processing:
read(0, "--- a/nodes/ui_base.html\n+++ b/n"..., 1024) = 1024
lseek(0, 0, SEEK_CUR) = 1024
lseek(0, 0, SEEK_CUR) = 1024
openat(AT_FDCWD, "nodes", O_RDONLY|O_LARGEFILE|O_NOFOLLOW|O_DIRECTORY) = 3
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1073741816, rlim_max=1073741816}) = 0
mmap(NULL, 5368709188, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Out of memory)
I've downloaded patch 2.7.6 sources and was able to reproduce the problem. From the stack trace it seems that building a hash table for the directory is the culprit:
Catchpoint 1 (returned from syscall mmap), __mmap (off=0, fd=-1, flags=34, prot=3, len=5368709188, start=0x0) at src/mman/mmap.c:34
34 in src/mman/mmap.c
(gdb) bt
#0 __mmap (off=0, fd=-1, flags=34, prot=3, len=5368709188, start=0x0) at src/mman/mmap.c:34
#1 __mmap (start=start@entry=0x0, len=len@entry=5368709188, prot=prot@entry=3, flags=flags@entry=34, fd=fd@entry=-1, off=off@entry=0) at src/mman/mmap.c:14
#2 0x0000fffff7f78dbc in __libc_malloc_impl (n=5368709168) at src/malloc/mallocng/malloc.c:310
#3 0x0000fffff7f77694 in calloc (m=m@entry=335544323, n=5368709168, n@entry=16) at src/malloc/calloc.c:40
#4 0x0000aaaaaaab6fa8 in hash_initialize (candidate=268435454, tuning=0xaaaaaaac5970 <default_tuning>, tuning@entry=0x0, hasher=hasher@entry=0xaaaaaaaaec54 <hash_cached_dirfd>, comparator=comparator@entry=0xaaaaaaaaec10 <compare_cached_dirfds>, data_freer=data_freer@entry=0x0) at hash.c:626
#5 0x0000aaaaaaaaee50 in init_dirfd_cache () at safe.c:105
#6 insert_cached_dirfd (entry=entry@entry=0xfffff7ffc9a0, keepfd=keepfd@entry=-1) at safe.c:149
#7 0x0000aaaaaaaaf0b0 in openat_cached (keepfd=-1, name=0xfffffffff0c0 "nodes", dir=0xaaaaaaae0068 <cwd>) at safe.c:240
#8 traverse_next (dir=dir@entry=0xaaaaaaae0068 <cwd>, path=0xfffffffff2a8, path@entry=0xaaaaaaae0730 <diff_type>, keepfd=-1, symlink=0xfffffffff2b0) at safe.c:387
#9 0x0000aaaaaaaaf458 in traverse_another_path (pathname=0xfffffffff650, pathname@entry=0xfffffffff2e0, keepfd=keepfd@entry=-1) at safe.c:468
#10 0x0000aaaaaaaaf950 in traverse_path (pathname=0xfffffffff2e0) at safe.c:544
#11 safe_xstat (flags=256, buf=0xfffffffff4b0, pathname=<optimized out>) at safe.c:554
#12 safe_lstat (pathname=<optimized out>, buf=0xfffffffff4b0) at safe.c:569
#13 0x0000aaaaaaab3340 in stat_file (filename=filename@entry=0xaaaaaaae0cd0 "nodes/ui_base.html", st=st@entry=0xfffffffff4b0) at util.c:1678
#14 0x0000aaaaaaaad798 in intuit_diff_type (need_header=<optimized out>, need_header@entry=true, p_file_type=0xfffffffff7cc) at pch.c:886
#15 0x0000aaaaaaaae15c in there_is_another_patch (need_header=true, file_type=file_type@entry=0xfffffffff7cc) at pch.c:238
#16 0x0000aaaaaaaa4dec in main (argc=<optimized out>, argv=<optimized out>) at patch.c:197
It seems that allocation coorelates with the first argument to hash_initialize
, which is 268435454 currently!
It seems that the problem has been recognized a while ago, and a fix is in the current patch sources, see: https://github.com/mirror/patch/commit/61d7788b83b302207a67b82786f4fd79e3538f30. So it seems that "RLIMIT_NOFILE is set to RLIM_INFINITY" is the case on Alpine Linux?
In any case, I've confirmed that this change passes 174287 to hash_initialize
and therefor just allocating ~3.4MB of memory. Can Alpine Linux add this patch?
Note: This has been noticed while debugging patch: **** out of memory
issue which cropped up when running the same software on a newer Linux kernel. It seems that newer Linux kernels have different over-committing behavior (from what I can see by this change). I can fix the problem by tweaking overcommit settings, but I'd rather prefer if patch would not allocate as much memory