Skip to content
Snippets Groups Projects

main/linux-firmware: compress firmware with zstd

Merged Newbyte requested to merge Newbyte/aports:compress-firmware into master

Uses the upstream install-zst target. ZSTD_CLEVEL=19 saves around 40 MB over the default (which is 3, I think).

  • Uncompressed: 1,0 GB
  • ZSTD_CLEVEL unset: 509,1 MB
  • ZSTD_CLEVEL=10: 494,4 MB
  • ZSTD_CLEVEL=19: 469,7 MB

xz with the default compression level was 445,8 MB, but it is slower to decompress and about as slow as ZSTD_CLEVEL=19 to compress. I didn't measure the exact time, but it felt similarly long. ZSTD_CLEVEL=10 and ZSTD_CLEVEL unset were fast to compress. These tests were conducted on an i5-6200U with 8 GB of RAM, so not the beefiest system.

Closes #15756 (closed)


TODO:

More things?

Edited by Newbyte

Merge request reports

Merge request pipeline #251621 skipped

Merge request pipeline skipped for 5c5824b3

Merged by Natanael CopaNatanael Copa 6 months ago (Aug 5, 2024 3:38pm UTC)

Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • **** assigned to @ncopa

    assigned to @ncopa

  • Newbyte added 1 commit

    added 1 commit

    • d3417ab7 - main/linux-firmware: compress firmware with zstd

    Compare with previous version

  • Newbyte added 1 commit

    added 1 commit

    • 8b3a20c9 - main/linux-firmware: compress firmware with zstd

    Compare with previous version

  • Newbyte added 1 commit

    added 1 commit

    • 7d3d08b3 - main/linux-firmware: compress firmware with zstd

    Compare with previous version

  • Newbyte added 1 commit

    added 1 commit

    • 1b460ec2 - main/linux-firmware: compress firmware with zstd

    Compare with previous version

  • Newbyte added 1 commit

    added 1 commit

    • 88e21912 - main/linux-firmware: compress firmware with zstd

    Compare with previous version

  • Newbyte changed the description

    changed the description

  • Author Guest

    Review would be nice even though this is a "draft".

  • Contributor

    kernels need to have zstd decompression support enabled

    this (CONFIG_DECOMPRESS_ZSTD) has i think been the case forever for -lts and -edge. for lts since maybe 03f6d63b (then later dropped in the refactored config since it's default), for edge since similarly old

    xz with the default compression level was 445,8 MB, but it is slower to decompress

    yes, decompression matters a lot here and zstd is the best option for that

  • Contributor

    actually, no. it needs CONFIG_FW_LOADER_COMPRESS* which is not even enabled presently

  • Newbyte mentioned in issue #15758 (closed)

    mentioned in issue #15758 (closed)

    • Resolved by Newbyte

      Hey everyone, Emil here o/

      I'm the person who added install-zst (+ xz) upstream and got that into Arch.

      Really glad to see more distros opting-in for the compression. Adding some details, for anyone interested:

      • yes, zstd -19 is analogous to xz in terms of compression speed and compressed file sizes, yet decompression is easily 2-4 times faster
      • if you are concerned about compression time you can set ZSTD_NBTHREADS as needed, it defaults to 1
      • files compressed with 20+ are not supported by the kernel

      Note: some firmware files* needs to stay uncompressed, they are annotated as "RawFile" in WHENCE. If you see omissions/bugs please send your patches upstream.

      Tangentially related: using zst compressed kernel modules will improve the boot times. Similarly to here compression should be capped at 19 - older kmod will decompress, while newer will pass the zst blob directly to the kernel.

      Looking at this MR itself: it looks solid, but one needs to enable CONFIG_FW_LOADER_COMPRESS_ZSTD for linux-virt.

      HTH

      Edited by Emil Velikov
    • Contributor
      Resolved by Newbyte

      something that is currently broken here i think is mkinitfs does https://github.com/alpinelinux/mkinitfs/blob/cd8e45f8c636a4c441ba7f95a289adcc80bef5b4/mkinitfs.in#L144

      the xargs modinfo -k $kernel -F firmware outputs something like

      amdgpu/polaris10_smc.bin
      amdgpu/polaris10_smc_sk.bin
      amdgpu/polaris10_k_smc.bin
      amdgpu/polaris10_k2_smc.bin

      etc. but the actual files on disk are not named that anymore. it has to take into account that it can have an extension.

      (this is hard to immediately notice was broken because it only affects things when you need the firmware to load the module and the module is also in the initrd (which isn't most things))

      for instance, if you put amdgpu in your initrd, then load the module early.. it will fail init without the firmware, and won't pick it up later at all (after root mount). so you would just have no gpu initialised (unless afaik you rmmod and probe it again)

      ubuntus patch on debian initramfs-tools does something like https://git.launchpad.net/ubuntu/+source/initramfs-tools/commit/hook-functions?id=100638d4cc5705fbb0bf70e1b6f81a25c1e43397 (previously ubuntu handled just xz for the old diff; vanilla debian initramfs handles nothing). note that it checks the plain name and then skips it later too (in case there is for some reason multiple compressions for the same file lol). in any case mkinitfs needs updating and a release to handle this, and postmarketos might also need checking for their initramfs generator, and there is also dracut/booster (didn't check any of them)

      Edited by alice
  • Newbyte added 1972 commits

    added 1972 commits

    Compare with previous version

  • Newbyte added 1 commit

    added 1 commit

    • c3b35ef6 - main/linux-firmware: compress firmware with zstd

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading