busybox xargs causes lbu to truncate file list when committing large number of files
On Alpine 2.4.3, tested busybox-1.19.4-r4 with alpine-conf-2.11.2-r0. Also tested busybox-1.20.1-r1.apk from edge repository.
With a large file list of >32768 bytes, I found that lbu would only commit the files in the list from approximately the first 32768 bytes.
Since http://git.alpinelinux.org/cgit/alpine-conf/commit/?id=4e857d18684667e7d9df3a0256e0b6e446284641, lbu provides the included file list via stdin to xargs, rather than a shell variable:
_gen_filelist_0 | xargs -0 $tar_create \ | gzip -c >"$tmppkg"
which expands to:
apk audit --backup --quiet --recursive | tr '\n' '\0' |\ xargs -0 tar -c --no-recursion | gzip -c >"$tmppkg"
Therefore this bug can now be triggered when using
lbu include to
store a large list of files.
To easily duplicate the fundamental problem in busybox xargs:
mkdir test cd test for a in `seq 1000 2000`; do echo 'hello' > aaaaabbbbbcccccdddddeeeeefffffggggghhhhh-$a done ls -1 | xargs tar -c --no-recursion | tar -t
At first I thought this commit might have been at fault http://git.busybox.net/busybox/commit/findutils/xargs.c?h=1\_19\_stable&id=4a96617095f40e3cfcf148a8a7d5d83f71079aa1, but I also tested Alpine 1.10 with busybox-1.16.2-r2 and duplicated the problem.
apk add findutils works around the problem on all systems
apk add tar or removing the —no-recursion option has no
effect on this problem.)
(from redmine: issue id 1240, created on 2012-07-03, closed on 2012-07-08)
- Revision 15245c4a0eb62b0e1bcbac5063391e0d57a53799 by Natanael Copa on 2012-07-04T08:09:21Z:
lbu: do not use xargs xargs will execute the command twice if the buffer becomes long enough. This will break tar. We also use -z instead of piping via gzip while here. ref #1240
- Revision 1ab89daa by Natanael Copa on 2012-07-04T15:07:45Z:
main/alpine-conf: lbu: do not use xargs ref #1240
- Revision b16dbb24 by Natanael Copa on 2012-07-04T15:09:30Z:
main/alpine-conf: lbu: do not use xargs fixes #1240 (cherry picked from commit 1ab89daa7c0d4b46230a2608f04151214a266101)