[2.3] squid 3.2 mem leak? (#1177) · Issues · alpine / aports

[2.3] squid 3.2 mem leak?

My squid boxes repeatedly stop working with the following error:

  
\[1131887.518529\] \[\] error\_code+0x76/0x80  
\[1131887.518535\] Mem-Info:  
\[1131887.518540\] DMA per-cpu:  
\[1131887.518546\] CPU 0: hi: 0, btch: 1 usd: 0  
\[1131887.518551\] Normal per-cpu:  
\[1131887.518557\] CPU 0: hi: 186, btch: 31 usd: 119  
\[1131887.518562\] HighMem per-cpu:  
\[1131887.518568\] CPU 0: hi: 42, btch: 7 usd: 6  
\[1131887.518582\] active\_anon:221759 inactive\_anon:13521
isolated\_anon:0  
\[1131887.518586\] active\_file:0 inactive\_file:0 isolated\_file:0  
\[1131887.518590\] unevictable:0 dirty:0 writeback:0 unstable:0  
\[1131887.518593\] free:13151 slab\_reclaimable:394
slab\_unreclaimable:1502  
\[1131887.518597\] mapped:2151 shmem:30504 pagetables:318 bounce:0  
\[1131887.518684\] DMA free:4724kB min:784kB low:980kB high:1176kB
active\_anon:11180kB inactive\_anon:0kB active\_file:0kB
inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:4kB
slab\_reclaimable:0kB slab\_unreclaimable:4kB kernel\_stack:0kB
pagetables:8kB unstable:0kB bounce:0kB writeback\_tmp:0kB
pages\_scanned:0 all\_unreclaimable? yes  
\[1131887.518702\] lowmem\_reserve\[\]: 0 865 983 983  
\[1131887.518724\] Normal free:47756kB min:44012kB low:55012kB
high:66016kB active\_anon:766644kB inactive\_anon:44836kB
active\_file:0kB inactive\_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:885944kB mlocked:0kB dirty:0kB writeback:0kB
mapped:7348kB shmem:56704kB slab\_reclaimable:1576kB
slab\_unreclaimable:6004kB kernel\_stack:376kB pagetables:1264kB
unstable:0kB bounce:0kB writeback\_tmp:0kB pages\_scanned:0
all\_unreclaimable? yes  
\[1131887.518742\] lowmem\_reserve\[\]: 0 0 943 943  
\[1131887.518763\] HighMem free:124kB min:128kB low:1628kB high:3128kB
active\_anon:109212kB inactive\_anon:9248kB active\_file:0kB
inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:120784kB mlocked:0kB dirty:0kB writeback:0kB mapped:1256kB
shmem:65308kB slab\_reclaimable:0kB slab\_unreclaimable:0kB
kernel\_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
writeback\_tmp:0kB pages\_scanned:0 all\_unreclaimable? yes  
\[1131887.518781\] lowmem\_reserve\[\]: 0 0 0 0  
\[1131887.518789\] DMA: 1\*4kB 0\*8kB 1\*16kB 1\*32kB 1\*64kB 0\*128kB
0\*256kB 1\*512kB 0\*1024kB 0\*2048kB 1\*4096kB = 4724kB  
\[1131887.518812\] Normal: 139\*4kB 103\*8kB 41\*16kB 47\*32kB 29\*64kB
19\*128kB 14\*256kB 7\*512kB 6\*1024kB 3\*2048kB 5\*4096kB = 47764kB  
\[1131887.518835\] HighMem: 3\*4kB 0\*8kB 1\*16kB 1\*32kB 1\*64kB
0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 124kB  
\[1131887.518857\] 30504 total pagecache pages  
\[1131887.518862\] 0 pages in swap cache  
\[1131887.518868\] Swap cache stats: add 0, delete 0, find 0/0  
\[1131887.518873\] Free swap = 0kB  
\[1131887.518877\] Total swap = 0kB  
\[1131887.536870\] 257744 pages RAM  
\[1131887.536877\] 30434 pages HighMem  
\[1131887.536881\] 3392 pages reserved  
\[1131887.536885\] 5306 pages shared  
\[1131887.536889\] 238848 pages non-shared  
\[1131887.536895\] \[ pid \] uid tgid total\_vm rss cpu oom\_adj
oom\_score\_adj name  
\[1131887.536922\] \[ 1493\] 0 1493 441 17 0 0 0 acpid  
\[1131887.536933\] \[ 3102\] 0 3102 227 123 0 0 0 chronyd  
\[1131887.536945\] \[ 3125\] 0 3125 434 17 0 0 0 klogd  
\[1131887.536957\] \[ 3137\] 0 3137 445 18 0 0 0 syslogd  
\[1131887.536968\] \[ 3149\] 0 3149 436 130 0 0 0 crond  
\[1131887.536980\] \[ 3197\] 1000 3197 1796 1464 0 0 0 lighttpd  
\[1131887.536992\] \[ 3244\] 0 3244 894 827 0 0 0 ucarp  
\[1131887.537004\] \[ 3261\] 0 3261 447 155 0 0 0 getty  
\[1131887.537016\] \[ 3262\] 0 3262 442 154 0 0 0 getty  
\[1131887.537027\] \[ 3265\] 0 3265 446 155 0 0 0 getty  
\[1131887.537039\] \[ 3268\] 0 3268 444 155 0 0 0 getty  
\[1131887.537051\] \[ 3271\] 0 3271 442 155 0 0 0 getty  
\[1131887.537063\] \[28952\] 0 28952 753 153 0 –17 –1000 sshd  
\[1131887.537074\] \[29020\] 0 29020 3225 379 0 0 0 squid  
\[1131887.537085\] \[21259\] 0 21259 206098 205235 0 0 0 squid  
\[1131887.537097\] \[21261\] 31 21261 157 26 0 0 0 unlinkd  
\[1131887.537109\] \[ 8771\] 31 8771 13399 146 0 0 0 squark-auth-ip  
\[1131887.537118\] Out of memory: Kill process 21259 (squid) score 807
or sacrifice child  
\[1131887.537129\] Killed process 21261 (unlinkd) total-vm:628kB,
anon-rss:32kB, file-rss:72kB  
\[1131887.540301\] squid invoked oom-killer: gfp\_mask=0x280da, order=0,
oom\_adj=0, oom\_score\_adj=0  
\[1131887.540314\] Pid: 21259, comm: squid Not tainted 3.0.18-grsec
\#2-Alpine  
\[1131887.540320\] Call Trace:  
\[1131887.540339\] \[\] dump\_header.isra.8+0x55/0x14f  
\[1131887.540355\] \[\] ? \_ratelimit+0xc6/0xd0  
\[1131887.540366\] \[\]
oom\_kill\_process.constprop.15+0x26/0x1dc  
\[1131887.540377\] \[\] ? has\_capability\_noaudit+0xf/0x16  
\[1131887.540387\] \[\] out\_of\_memory+0x218/0x27f  
\[1131887.540398\] \[\]
\_\_alloc\_pages\_nodemask+0x448/0x50b  
\[1131887.540415\] \[\] handle\_pte\_fault+0xda/0x897  
\[1131887.540427\] \[\] ? spurious\_fault+0xb6/0xb6  
\[1131887.540436\] \[\] handle\_mm\_fault+0x2de/0x2f4  
\[1131887.540445\] \[\] ? spurious\_fault+0xb6/0xb6  
\[1131887.540454\] \[\] do\_page\_fault+0x53e/0x559  
\[1131887.540464\] \[\] ? do\_brk+0x262/0x27f  
\[1131887.540473\] \[\] ? sys\_brk+0xf1/0xfb  
\[1131887.540481\] \[\] ? spurious\_fault+0xb6/0xb6  
\[1131887.540489\] \[\] error\_code+0x76/0x80  
\[1131887.540495\] Mem-Info:  
\[1131887.540499\] DMA per-cpu:  
\[1131887.540505\] CPU 0: hi: 0, btch: 1 usd: 0  
\[1131887.540510\] Normal per-cpu:  
\[1131887.540516\] CPU 0: hi: 186, btch: 31 usd: 125  
\[1131887.540521\] HighMem per-cpu:  
\[1131887.540527\] CPU 0: hi: 42, btch: 7 usd: 17  
\[1131887.540541\] active\_anon:221750 inactive\_anon:13521
isolated\_anon:0  
\[1131887.540545\] active\_file:0 inactive\_file:0 isolated\_file:0  
\[1131887.540548\] unevictable:0 dirty:0 writeback:0 unstable:0  
\[1131887.540552\] free:13147 slab\_reclaimable:394
slab\_unreclaimable:1506  
\[1131887.540556\] mapped:2151 shmem:30504 pagetables:305 bounce:0  
\[1131887.540576\] DMA free:4708kB min:784kB low:980kB high:1176kB
active\_anon:11180kB inactive\_anon:0kB active\_file:0kB
inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:4kB
slab\_reclaimable:0kB slab\_unreclaimable:20kB kernel\_stack:0kB
pagetables:8kB unstable:0kB bounce:0kB writeback\_tmp:0kB
pages\_scanned:0 all\_unreclaimable? yes  
\[1131887.540593\] lowmem\_reserve\[\]: 0 865 983 983  
\[1131887.540615\] Normal free:47756kB min:44012kB low:55012kB
high:66016kB active\_anon:766644kB inactive\_anon:44836kB
active\_file:0kB inactive\_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:885944kB mlocked:0kB dirty:0kB writeback:0kB
mapped:7348kB shmem:56704kB slab\_reclaimable:1576kB
slab\_unreclaimable:6004kB kernel\_stack:376kB pagetables:1212kB
unstable:0kB bounce:0kB writeback\_tmp:0kB pages\_scanned:0
all\_unreclaimable? yes  
\[1131887.540634\] lowmem\_reserve\[\]: 0 0 943 943  
\[1131887.540655\] HighMem free:124kB min:128kB low:1628kB high:3128kB
active\_anon:109176kB inactive\_anon:9248kB active\_file:0kB
inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:120784kB mlocked:0kB dirty:0kB writeback:0kB mapped:1256kB
shmem:65308kB slab\_reclaimable:0kB slab\_unreclaimable:0kB
kernel\_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
writeback\_tmp:0kB pages\_scanned:0 all\_unreclaimable? yes  
\[1131887.540673\] lowmem\_reserve\[\]: 0 0 0 0  
\[1131887.540681\] DMA: 0\*4kB 0\*8kB 0\*16kB 1\*32kB 1\*64kB 0\*128kB
0\*256kB 1\*512kB 0\*1024kB 0\*2048kB 1\*4096kB = 4704kB  
\[1131887.540702\] Normal: 139\*4kB 103\*8kB 41\*16kB 47\*32kB 29\*64kB
19\*128kB 14\*256kB 7\*512kB 6\*1024kB 3\*2048kB 5\*4096kB = 47764kB  
\[1131887.540725\] HighMem: 3\*4kB 0\*8kB 1\*16kB 1\*32kB 1\*64kB
0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 124kB  
\[1131887.540748\] 30504 total pagecache pages  
\[1131887.540753\] 0 pages in swap cache  
\[1131887.540758\] Swap cache stats: add 0, delete 0, find 0/0  
\[1131887.540763\] Free swap = 0kB  
\[1131887.540767\] Total swap = 0kB  
\[1131887.558868\] 257744 pages RAM  
\[1131887.558875\] 30434 pages HighMem  
\[1131887.558879\] 3392 pages reserved  
\[1131887.558883\] 5246 pages shared  
\[1131887.558888\] 238837 pages non-shared  
\[1131887.558893\] \[ pid \] uid tgid total\_vm rss cpu oom\_adj
oom\_score\_adj name  
\[1131887.558920\] \[ 1493\] 0 1493 441 17 0 0 0 acpid  
\[1131887.558932\] \[ 3102\] 0 3102 227 123 0 0 0 chronyd  
\[1131887.558944\] \[ 3125\] 0 3125 434 17 0 0 0 klogd  
\[1131887.558955\] \[ 3137\] 0 3137 445 18 0 0 0 syslogd  
\[1131887.558967\] \[ 3149\] 0 3149 436 130 0 0 0 crond  
\[1131887.558979\] \[ 3197\] 1000 3197 1796 1464 0 0 0 lighttpd  
\[1131887.558990\] \[ 3244\] 0 3244 894 827 0 0 0 ucarp  
\[1131887.559002\] \[ 3261\] 0 3261 447 155 0 0 0 getty  
\[1131887.559013\] \[ 3262\] 0 3262 442 154 0 0 0 getty  
\[1131887.559025\] \[ 3265\] 0 3265 446 155 0 0 0 getty  
\[1131887.559036\] \[ 3268\] 0 3268 444 155 0 0 0 getty  
\[1131887.559048\] \[ 3271\] 0 3271 442 155 0 0 0 getty  
\[1131887.559059\] \[28952\] 0 28952 753 153 0 –17 –1000 sshd  
\[1131887.559071\] \[29020\] 0 29020 3225 379 0 0 0 squid  
\[1131887.559083\] \[21259\] 0 21259 206098 205235 0 0 0 squid  
\[1131887.559095\] \[ 8771\] 31 8771 13399 146 0 0 0 squark-auth-ip  
\[1131887.559104\] Out of memory: Kill process 21259 (squid) score 807
or sacrifice child  
\[1131887.559115\] Killed process 8771 (squark-auth-ip)
total-vm:53596kB, anon-rss:84kB, file-rss:500kB  
\[1131887.562860\] squid invoked oom-killer: gfp\_mask=0x280da, order=0,
oom\_adj=0, oom\_score\_adj=0  
\[1131887.562874\] Pid: 21259, comm: squid Not tainted 3.0.18-grsec
\#2-Alpine  
\[1131887.562880\] Call Trace:  
\[1131887.562900\] \[\] dump\_header.isra.8+0x55/0x14f  
\[1131887.562915\] \[\] ? \_ratelimit+0xc6/0xd0  
\[1131887.562926\] \[\]
oom\_kill\_process.constprop.15+0x26/0x1dc  
\[1131887.562937\] \[\] ? has\_capability\_noaudit+0xf/0x16  
\[1131887.562946\] \[\] out\_of\_memory+0x218/0x27f  
\[1131887.562957\] \[\]
\_\_alloc\_pages\_nodemask+0x448/0x50b  
\[1131887.562974\] \[\] handle\_pte\_fault+0xda/0x897  
\[1131887.562986\] \[\] ? spurious\_fault+0xb6/0xb6  
\[1131887.562996\] \[\] handle\_mm\_fault+0x2de/0x2f4  
\[1131887.563005\] \[\] ? spurious\_fault+0xb6/0xb6  
\[1131887.563013\] \[\] do\_page\_fault+0x53e/0x559  
\[1131887.563023\] \[\] ? do\_brk+0x262/0x27f  
\[1131887.563032\] \[\] ? sys\_brk+0xf1/0xfb  
\[1131887.563040\] \[\] ? spurious\_fault+0xb6/0xb6  
\[1131887.563048\] \[\] error\_code+0x76/0x80  
\[1131887.563054\] Mem-Info:  
\[1131887.563059\] DMA per-cpu:  
\[1131887.563065\] CPU 0: hi: 0, btch: 1 usd: 0  
\[1131887.563070\] Normal per-cpu:  
\[1131887.563075\] CPU 0: hi: 186, btch: 31 usd: 142  
\[1131887.563080\] HighMem per-cpu:  
\[1131887.563086\] CPU 0: hi: 42, btch: 7 usd: 39  
\[1131887.563100\] active\_anon:221726 inactive\_anon:13521
isolated\_anon:0  
\[1131887.563104\] active\_file:0 inactive\_file:0 isolated\_file:0  
\[1131887.563108\] unevictable:0 dirty:0 writeback:0 unstable:0  
\[1131887.563111\] free:13133 slab\_reclaimable:394
slab\_unreclaimable:1520  
\[1131887.563115\] mapped:2097 shmem:30504 pagetables:292 bounce:0  
\[1131887.563135\] DMA free:4652kB min:784kB low:980kB high:1176kB
active\_anon:11180kB inactive\_anon:0kB active\_file:0kB
inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:4kB
slab\_reclaimable:0kB slab\_unreclaimable:76kB kernel\_stack:0kB
pagetables:8kB unstable:0kB bounce:0kB writeback\_tmp:0kB
pages\_scanned:0 all\_unreclaimable? yes  
\[1131887.563153\] lowmem\_reserve\[\]: 0 865 983 983  
\[1131887.563175\] Normal free:47756kB min:44012kB low:55012kB
high:66016kB active\_anon:766644kB inactive\_anon:44836kB
active\_file:0kB inactive\_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:885944kB mlocked:0kB dirty:0kB writeback:0kB
mapped:7244kB shmem:56704kB slab\_reclaimable:1576kB
slab\_unreclaimable:6004kB kernel\_stack:376kB pagetables:1160kB
unstable:0kB bounce:0kB writeback\_tmp:0kB pages\_scanned:0
all\_unreclaimable? yes  
\[1131887.563193\] lowmem\_reserve\[\]: 0 0 943 943  
\[1131887.563214\] HighMem free:124kB min:128kB low:1628kB high:3128kB
active\_anon:109080kB inactive\_anon:9248kB active\_file:0kB
inactive\_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:120784kB mlocked:0kB dirty:0kB writeback:0kB mapped:1144kB
shmem:65308kB slab\_reclaimable:0kB slab\_unreclaimable:0kB
kernel\_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
writeback\_tmp:0kB pages\_scanned:0 all\_unreclaimable? yes  
\[1131887.563232\] lowmem\_reserve\[\]: 0 0 0 0  
\[1131887.563240\] DMA: 1\*4kB 0\*8kB 0\*16kB 1\*32kB 0\*64kB 0\*128kB
0\*256kB 1\*512kB 0\*1024kB 0\*2048kB 1\*4096kB = 4644kB  
\[1131887.563262\] Normal: 139\*4kB 103\*8kB 41\*16kB 47\*32kB 29\*64kB
19\*128kB 14\*256kB 7\*512kB 6\*1024kB 3\*2048kB 5\*4096kB = 47764kB  
\[1131887.563285\] HighMem: 3\*4kB 0\*8kB 1\*16kB 1\*32kB 1\*64kB
0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 124kB  
\[1131887.563307\] 30504 total pagecache pages  
\[1131887.563312\] 0 pages in swap cache  
\[1131887.563318\] Swap cache stats: add 0, delete 0, find 0/0  
\[1131887.563323\] Free swap = 0kB  
\[1131887.563327\] Total swap = 0kB  
\[1131887.580694\] 257744 pages RAM  
\[1131887.580700\] 30434 pages HighMem  
\[1131887.580704\] 3392 pages reserved  
\[1131887.580708\] 5076 pages shared  
\[1131887.580712\] 238859 pages non-shared  
\[1131887.580718\] \[ pid \] uid tgid total\_vm rss cpu oom\_adj
oom\_score\_adj name  
\[1131887.580744\] \[ 1493\] 0 1493 441 17 0 0 0 acpid  
\[1131887.580756\] \[ 3102\] 0 3102 227 123 0 0 0 chronyd  
\[1131887.580768\] \[ 3125\] 0 3125 434 17 0 0 0 klogd  
\[1131887.580780\] \[ 3137\] 0 3137 445 18 0 0 0 syslogd  
\[1131887.580791\] \[ 3149\] 0 3149 436 130 0 0 0 crond  
\[1131887.580803\] \[ 3197\] 1000 3197 1796 1464 0 0 0 lighttpd  
\[1131887.580814\] \[ 3244\] 0 3244 894 827 0 0 0 ucarp  
\[1131887.580826\] \[ 3261\] 0 3261 447 155 0 0 0 getty  
\[1131887.580838\] \[ 3262\] 0 3262 442 154 0 0 0 getty  
\[1131887.580849\] \[ 3265\] 0 3265 446 155 0 0 0 getty  
\[1131887.580861\] \[ 3268\] 0 3268 444 155 0 0 0 getty  
\[1131887.580872\] \[ 3271\] 0 3271 442 155 0 0 0 getty  
\[1131887.580884\] \[28952\] 0 28952 753 153 0 –17 –1000 sshd  
\[1131887.580896\] \[29020\] 0 29020 3225 379 0 0 0 squid  
\[1131887.580907\] \[21259\] 0 21259 206098 205235 0 0 0 squid  
\[1131887.580917\] Out of memory: Kill process 21259 (squid) score 807
or sacrifice child  
\[1131887.580927\] Killed process 21259 (squid) total-vm:824392kB,
anon-rss:812628kB, file-rss:8312kB

They exhibit this behavior after few days they are running.
The boxes have 1GB RAM running from USB (webcaching is disabled, running in “normal” mode, no “intercepting/transparent”):

proxy2:~# uname -a
Linux proxy2 3.0.18-grsec #2 (closed)-Alpine SMP Mon Jan 30 16:04:05 UTC 2012 i686 Linux
proxy2:~# cat /etc/alpine-release
2.3.6
proxy2:~# apk version squid
Installed: Available:
squid-3.2.0.13-r0 = 3.2.0.13-r0

I think I’ll try upgrade one box to 2.4 and see what happens.

(from redmine: issue id 1177, created on 2012-05-23, closed on 2012-05-24)

Relations:
- duplicates #1054 (closed)
Changesets:
- Revision 5b265f1e by Timo Teräs on 2012-05-23T11:47:56Z:

main/squid: fix a memory leak

fixes #1054
fixes #1177

Fix authored by me; and reported also to upstream:
http://bugs.squid-cache.org/show_bug.cgi?id=3537
(cherry picked from commit 318655679724213db24a9da9d5a5ddba3efe2254)

To upload designs, you'll need to enable LFS and have admin enable hashed storage. More information