apk is perceivably slower in an LXC container than in the "global zone" on the same machine
I know this report is quite vague, but the issue is pretty annoying and I'm not sure where to look for further clues. I searched in Gitlab, but could not find any relevant issues.
I run a small home server on a PC Engines APU2C4 box. The global zone is Alpine x86_64 3.14, and there's also an LXC container with the same OS. The server runs some other workloads, but it's not really heavily loaded.
Both Alpine deployments use the same remote package repository (dl-4) and the same HTTP proxy. There's only one restriction on the LXC container: the memory is capped at 128 MB, otherwise it can use all the available 4 CPU cores (just like the global zone). But there's plenty of free memory, so it should not be a limiting factor.
The GZ uses Ext4, but the container is on ZFS (same mSATA SSD, different partitions). Watching apk with
strace revealed it does some heavy lifting in
/lib/apk/db, so to give an edge to the LXC container, I moved it temporarily to ramdisk. It did not help.
After having installed
ltrace, I measured the library calls, so here are some numbers below. I executed
ltrace -t apk install ltrace a couple of times on each OS instances, which stresses the package manager, but does not change the system.
Snippet from the output in the global zone:
23:42:51 getopt_long(3, 0x7fff1f9630d8, "fhiqX:p:UvVslut:", 0x7fff1f9624b8, nil) = -1 23:42:51 strcmp("add", "add") = 0 23:42:51 apk_db_init(0x5633f983b900, 0x5633f983653a, 0, 4) = 0x7fac08a12790 23:42:52 signal(SIGINT, 0x5633f982ef29) = 0 23:42:52 apk_db_open(0x5633f983b900, 0x7fff1f963010, 0, 0) = 0 23:42:54 apk_array_resize(0x7fac08a12790, 1, 8, 179) = 0x7fac074ed200 23:42:54 apk_array_resize(0, 108, 24, 0) = 0x7fac074a2ab0 23:42:54 getuid() = 0
The LXC container, when
/lib/apk/db is on ZFS (compression=lz4, atime=off):
23:42:49 getopt_long(3, 0x7fff74767118, "fhiqX:p:UvVslut:", 0x7fff747664f8, nil) = -1 23:42:49 strcmp("add", "add") = 0 23:42:49 apk_db_init(0x555f33c82900, 0x555f33c7d53a, 0, 4) = 0x7fedae2ce790 23:42:49 signal(SIGINT, 0x555f33c75f29) = 0 23:42:49 apk_db_open(0x555f33c82900, 0x7fff74767050, 0, 0) = 0 23:42:55 apk_array_resize(0x7fedae2ce790, 1, 8, 179) = 0x7fedac65cdd0 23:42:55 apk_array_resize(0, 72, 24, 0) = 0x7fedac798f10 23:42:55 getuid() = 0
The LXC container, now
/lib/apk/db was tmpfs (original files copied over):
23:49:50 getopt_long(3, 0x7ffd23e96b48, "fhiqX:p:UvVslut:", 0x7ffd23e95f28, nil) = -1 23:49:50 strcmp("add", "add") = 0 23:49:50 apk_db_init(0x559689884900, 0x55968987f53a, 0, 4) = 0x7f066576e790 23:49:50 signal(SIGINT, 0x559689877f29) = 0 23:49:50 apk_db_open(0x559689884900, 0x7ffd23e96a80, 0, 0) = 0 23:49:55 apk_array_resize(0x7f066576e790, 1, 8, 179) = 0x7f0663afcdd0 23:49:55 apk_array_resize(0, 72, 24, 0) = 0x7f0663c38f10 23:49:55 getuid() = 0
As you can see,
apk_db_open() suffers a massive, 5-6s delay, compared to the instance running in the global zone.
TBH the container has more packages installed, the
/lib/apk/db/installed file is 1064 kB in the GZ, but 4806 kB in the LXC container (2401 kB on disk because of ZFS compression).
I would not think this difference in the file size causes the delay, or does it?
Any clues would be appreciated, or tips what other tests I should run.