apk is perceivably slower in an LXC container than in the "global zone" on the same machine
Hello,
I know this report is quite vague, but the issue is pretty annoying and I'm not sure where to look for further clues. I searched in Gitlab, but could not find any relevant issues.
I run a small home server on a PC Engines APU2C4 box. The global zone is Alpine x86_64 3.14, and there's also an LXC container with the same OS. The server runs some other workloads, but it's not really heavily loaded.
Both Alpine deployments use the same remote package repository (dl-4) and the same HTTP proxy. There's only one restriction on the LXC container: the memory is capped at 128 MB, otherwise it can use all the available 4 CPU cores (just like the global zone). But there's plenty of free memory, so it should not be a limiting factor.
The GZ uses Ext4, but the container is on ZFS (same mSATA SSD, different partitions). Watching apk with strace
revealed it does some heavy lifting in /lib/apk/db
, so to give an edge to the LXC container, I moved it temporarily to ramdisk. It did not help.
After having installed ltrace
, I measured the library calls, so here are some numbers below. I executed ltrace -t apk install ltrace
a couple of times on each OS instances, which stresses the package manager, but does not change the system.
Snippet from the output in the global zone:
23:42:51 getopt_long(3, 0x7fff1f9630d8, "fhiqX:p:UvVslut:", 0x7fff1f9624b8, nil) = -1
23:42:51 strcmp("add", "add") = 0
23:42:51 apk_db_init(0x5633f983b900, 0x5633f983653a, 0, 4) = 0x7fac08a12790
23:42:52 signal(SIGINT, 0x5633f982ef29) = 0
23:42:52 apk_db_open(0x5633f983b900, 0x7fff1f963010, 0, 0) = 0
23:42:54 apk_array_resize(0x7fac08a12790, 1, 8, 179) = 0x7fac074ed200
23:42:54 apk_array_resize(0, 108, 24, 0) = 0x7fac074a2ab0
23:42:54 getuid() = 0
The LXC container, when /lib/apk/db
is on ZFS (compression=lz4, atime=off):
23:42:49 getopt_long(3, 0x7fff74767118, "fhiqX:p:UvVslut:", 0x7fff747664f8, nil) = -1
23:42:49 strcmp("add", "add") = 0
23:42:49 apk_db_init(0x555f33c82900, 0x555f33c7d53a, 0, 4) = 0x7fedae2ce790
23:42:49 signal(SIGINT, 0x555f33c75f29) = 0
23:42:49 apk_db_open(0x555f33c82900, 0x7fff74767050, 0, 0) = 0
23:42:55 apk_array_resize(0x7fedae2ce790, 1, 8, 179) = 0x7fedac65cdd0
23:42:55 apk_array_resize(0, 72, 24, 0) = 0x7fedac798f10
23:42:55 getuid() = 0
The LXC container, now /lib/apk/db
was tmpfs (original files copied over):
23:49:50 getopt_long(3, 0x7ffd23e96b48, "fhiqX:p:UvVslut:", 0x7ffd23e95f28, nil) = -1
23:49:50 strcmp("add", "add") = 0
23:49:50 apk_db_init(0x559689884900, 0x55968987f53a, 0, 4) = 0x7f066576e790
23:49:50 signal(SIGINT, 0x559689877f29) = 0
23:49:50 apk_db_open(0x559689884900, 0x7ffd23e96a80, 0, 0) = 0
23:49:55 apk_array_resize(0x7f066576e790, 1, 8, 179) = 0x7f0663afcdd0
23:49:55 apk_array_resize(0, 72, 24, 0) = 0x7f0663c38f10
23:49:55 getuid() = 0
As you can see, apk_db_open()
suffers a massive, 5-6s delay, compared to the instance running in the global zone.
TBH the container has more packages installed, the /lib/apk/db/installed
file is 1064 kB in the GZ, but 4806 kB in the LXC container (2401 kB on disk because of ZFS compression).
I would not think this difference in the file size causes the delay, or does it?
Any clues would be appreciated, or tips what other tests I should run.