main/linux-firmware: compress firmware with zstd
Uses the upstream install-zst target. ZSTD_CLEVEL=19 saves around 40 MB over the default (which is 3, I think).
- Uncompressed: 1,0 GB
- ZSTD_CLEVEL unset: 509,1 MB
- ZSTD_CLEVEL=10: 494,4 MB
- ZSTD_CLEVEL=19: 469,7 MB
xz with the default compression level was 445,8 MB, but it is slower to decompress and about as slow as ZSTD_CLEVEL=19 to compress. I didn't measure the exact time, but it felt similarly long. ZSTD_CLEVEL=10 and ZSTD_CLEVEL unset were fast to compress. These tests were conducted on an i5-6200U with 8 GB of RAM, so not the beefiest system.
Closes #15756 (closed)
TODO:
-
postmarketos-mkinitfs support: https://gitlab.com/postmarketOS/postmarketos-mkinitfs/-/merge_requests/49 -
(Alpine) mkinitfs support: mkinitfs!164 (merged) -
New (Alpine) mkinitfs release: mkinitfs@f95e5508 -
New (Alpine) mkinitfs release merged in Alpine: b214163a -
pd-mapper support, pd-mapper moved into kernel, or ".jsn" files marked as uncompressable upstream: All ".jsn" files are marked as RawFile in WHENCE: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/WHENCE -
tqftpserv support, or relevant firmware marked as uncompressable upstream: https://github.com/linux-msm/tqftpserv/pull/20 -
New tqftpserv release: https://github.com/linux-msm/tqftpserv/releases/tag/v1.1 -
New tqftpserv release merged in pmOS: https://gitlab.com/postmarketOS/pmaports/-/merge_requests/5370 -
CONFIG_FW_LOADER_COMPRESS_ZSTD
enabled in linux-lts: 8ff17746 -
CONFIG_FW_LOADER_COMPRESS_ZSTD
enabled in linux-edge: 492db628 -
CONFIG_FW_LOADER_COMPRESS_ZSTD
enabled in linux-rpi 6e1b700e -
CONFIG_FW_LOADER_COMPRESS_ZSTD
enabled in linux-virt 70507108 -
CONFIG_FW_LOADER_COMPRESS_ZSTD
enabled in linux-openpax 47451f18
More things?
Merge request reports
Activity
assigned to @ncopa
added 1 commit
- d3417ab7 - main/linux-firmware: compress firmware with zstd
added 1 commit
- 8b3a20c9 - main/linux-firmware: compress firmware with zstd
added 1 commit
- 7d3d08b3 - main/linux-firmware: compress firmware with zstd
added 1 commit
- 1b460ec2 - main/linux-firmware: compress firmware with zstd
added 1 commit
- 88e21912 - main/linux-firmware: compress firmware with zstd
kernels need to have zstd decompression support enabled
this (CONFIG_DECOMPRESS_ZSTD) has i think been the case forever for -lts and -edge. for lts since maybe 03f6d63b (then later dropped in the refactored config since it's default), for edge since similarly old
xz with the default compression level was 445,8 MB, but it is slower to decompress
yes, decompression matters a lot here and zstd is the best option for that
mentioned in issue #15758 (closed)
- Resolved by Newbyte
Hey everyone, Emil here o/
I'm the person who added install-zst (+ xz) upstream and got that into Arch.
Really glad to see more distros opting-in for the compression. Adding some details, for anyone interested:
- yes, zstd -19 is analogous to xz in terms of compression speed and compressed file sizes, yet decompression is easily 2-4 times faster
- if you are concerned about compression time you can set
ZSTD_NBTHREADS
as needed, it defaults to 1 - files compressed with 20+ are not supported by the kernel
Note: some firmware files* needs to stay uncompressed, they are annotated as "RawFile" in
WHENCE
. If you see omissions/bugs please send your patches upstream.Tangentially related: using zst compressed kernel modules will improve the boot times. Similarly to here compression should be capped at 19 - older kmod will decompress, while newer will pass the zst blob directly to the kernel.
Looking at this MR itself: it looks solid, but one needs to enable
CONFIG_FW_LOADER_COMPRESS_ZSTD
for linux-virt.HTH
Edited by Emil Velikov
- Resolved by Newbyte
something that is currently broken here i think is mkinitfs does https://github.com/alpinelinux/mkinitfs/blob/cd8e45f8c636a4c441ba7f95a289adcc80bef5b4/mkinitfs.in#L144
the
xargs modinfo -k $kernel -F firmware
outputs something likeamdgpu/polaris10_smc.bin amdgpu/polaris10_smc_sk.bin amdgpu/polaris10_k_smc.bin amdgpu/polaris10_k2_smc.bin
etc. but the actual files on disk are not named that anymore. it has to take into account that it can have an extension.
(this is hard to immediately notice was broken because it only affects things when you need the firmware to load the module and the module is also in the initrd (which isn't most things))
for instance, if you put amdgpu in your initrd, then load the module early.. it will fail init without the firmware, and won't pick it up later at all (after root mount). so you would just have no gpu initialised (unless afaik you rmmod and probe it again)
ubuntus patch on debian initramfs-tools does something like https://git.launchpad.net/ubuntu/+source/initramfs-tools/commit/hook-functions?id=100638d4cc5705fbb0bf70e1b6f81a25c1e43397 (previously ubuntu handled just xz for the old diff; vanilla debian initramfs handles nothing). note that it checks the plain name and then skips it later too (in case there is for some reason multiple compressions for the same file lol). in any case mkinitfs needs updating and a release to handle this, and postmarketos might also need checking for their initramfs generator, and there is also dracut/booster (didn't check any of them)
Edited by alice
- Resolved by Newbyte
Another blocker is fixing postmarketos-mkinitfs (downstream in postmarketOS).
added 1972 commits
-
88e21912...80179597 - 1971 commits from branch
alpine:master
- 94658290 - main/linux-firmware: compress firmware with zstd
-
88e21912...80179597 - 1971 commits from branch
added 1 commit
- c3b35ef6 - main/linux-firmware: compress firmware with zstd