apk-tools issueshttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues2024-03-11T08:31:37Zhttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10964Provide development documentation2024-03-11T08:31:37ZSander MaijersProvide development documentation[`docs/`](docs/) contains user manuals, but short of that, development documentation seems absent.
@ariadne wrote interesting analyses such as https://ariadne.space/2021/10/31/spelunking-through-the-apk-tools-dependency-solver/, which I ...[`docs/`](docs/) contains user manuals, but short of that, development documentation seems absent.
@ariadne wrote interesting analyses such as https://ariadne.space/2021/10/31/spelunking-through-the-apk-tools-dependency-solver/, which I think deserve a more central place.
In addition, personally, I'd like to learn about `apk-tools`'s design principles and architectural requirements. For example, is `/var/cache/apk/` supposed to be data race safe in the face of parallel `apk` processes?backloghttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10907apk policy listing packages in wrong order2024-03-08T16:21:40ZEdward Peekapk policy listing packages in wrong orderPer `apk policy --help` packages are supposed to be listed "in order of installation preference".
In Alpine 3.18 with `apk-tools-2.14.0-r2` I am instead observing multiple versions of the same package from a single custom repo being lis...Per `apk policy --help` packages are supposed to be listed "in order of installation preference".
In Alpine 3.18 with `apk-tools-2.14.0-r2` I am instead observing multiple versions of the same package from a single custom repo being listed unsorted in whatever order they are defined in the repo index. The first result is not what is installed:
```
# apk policy libuuid
libuuid policy:
2.38.1-r7:
https://.../v3.18/main
2.38.1-r8:
https://.../v3.18/main
/ # apk add libuuid
WARNING: This apk-tools is OLD! Some packages might not function properly.
(1/1) Installing libuuid (2.38.1-r8)
OK: 7 MiB in 16 packages
```
The repo `APKINDEX` contains:
```
P:libuuid
V:2.38.1-r7
A:x86_64
S:13571
I:40960
T:DCE compatible Universally Unique Identifier library
U:https://git.kernel.org/cgit/utils/util-linux/util-linux.git
L:BSD-3-Clause
o:util-linux
m:Natanael Copa <ncopa@alpinelinux.org>
t:1681557467
c:25bd220db5fce58966f17cf7719f9c73a5c5295e
D:so:libc.musl-x86_64.so.1
p:so:libuuid.so.1=1.3.0
z:v3.18/main/x86_64/libuuid-2.38.1-r7.apk
y:fbcdcb907098b719a1a7dd21aa48ddce83e8b62c
P:libuuid
V:2.38.1-r8
A:x86_64
S:13567
I:40960
T:DCE compatible Universally Unique Identifier library
U:https://git.kernel.org/cgit/utils/util-linux/util-linux.git
L:BSD-3-Clause
o:util-linux
m:Natanael Copa <ncopa@alpinelinux.org>
t:1686107202
c:c7de7fac9ae57f268781a733984e74a36f867d1c
D:so:libc.musl-x86_64.so.1
p:so:libuuid.so.1=1.3.0
```v3.0https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10895ERROR: script exited with error 02024-03-08T16:21:40ZPatrycja Rosaalpine@ptrcnull.meERROR: script exited with error 0in some weird cases, like running an arm64v8 container on x86_64 with qemu-user, apk can return `script exited with error 0`:
```console
$ podman run --rm -it arm64v8/alpine:3.15
WARNING: image platform (linux/arm64/v8) does not match t...in some weird cases, like running an arm64v8 container on x86_64 with qemu-user, apk can return `script exited with error 0`:
```console
$ podman run --rm -it arm64v8/alpine:3.15
WARNING: image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)
/ # sed -i 's/3.15/3.12/' /etc/apk/repositories
/ # apk upgrade -a
fetch https://dl-cdn.alpinelinux.org/alpine/v3.12/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.12/community/aarch64/APKINDEX.tar.gz
(1/15) Downgrading musl (1.2.2-r8 -> 1.1.24-r10)
(2/15) Downgrading busybox (1.34.1-r7 -> 1.31.1-r22)
Executing busybox-1.31.1-r22.post-upgrade
(3/15) Downgrading alpine-baselayout (3.2.0-r18 -> 3.2.0-r7)
Executing alpine-baselayout-3.2.0-r7.pre-upgrade
ERROR: alpine-baselayout-3.2.0-r7.pre-upgrade: script exited with error 0
Executing alpine-baselayout-3.2.0-r7.post-upgrade
(4/15) Downgrading alpine-keys (2.4-r1 -> 2.4-r0)
```
i narrowed it down to this line:
https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/v2.12.7/src/database.c#L1979
but couldn't find in which cases `WIFEXITED` would return a zero valuehttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10685Exposing security information in apk_package2024-03-08T14:45:40ZRasmus Thomsenoss@cogitri.devExposing security information in apk_packageIt would be nice if apk exposed information about fixed CVEs in a package upgrade so that users could better gauge how important an upgrade is (and users of libapk can set the urgency of an upgrade, e.g. to underline it with red colour t...It would be nice if apk exposed information about fixed CVEs in a package upgrade so that users could better gauge how important an upgrade is (and users of libapk can set the urgency of an upgrade, e.g. to underline it with red colour to show the upgrade is security relevant).v3.1https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10868add vendor field to apkv2/apkv3 control data and apkdb2024-03-08T14:45:39ZAriadne Conillariadne@ariadne.spaceadd vendor field to apkv2/apkv3 control data and apkdbSoftware component analysis tools (and related software such as vulnerability scanning tools) use CPE identifiers and Package URLs (PURLs) to determine the ecosystem a software component originates from.
At present, the apkdb does not c...Software component analysis tools (and related software such as vulnerability scanning tools) use CPE identifiers and Package URLs (PURLs) to determine the ecosystem a software component originates from.
At present, the apkdb does not contain this information, so SCA tools assume that all packages originate from the distribution. This causes problems when trying to understand the relationship between software components and their suppliers, to do things like find out about remediated CVEs, as the calculated CPE identifiers and PURLs are matched to the wrong supplier.
Since Alpine 3.16, we have shipped `/etc/secfixes.d/$vendorlabel`, e.g. `/etc/secfixes.d/alpine`, files in the baselayout. Some other apk distributions which use the same secdb format have done the same such as $dayjob's Wolfi GNU/Linux distribution. By providing a vendor tag in the APK control data and apkdb which matches to these `/etc/secfixes.d` files, we can correctly map relevant security remediation feeds to the packages.
I'm happy to work on this if @fabled agrees we should do this.v3.0https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10875add PURL field to apkv3 package data2024-03-08T14:45:39ZAriadne Conillariadne@ariadne.spaceadd PURL field to apkv3 package dataSecurity scanners have issues with disambiguating distribution package names (which exist in a flat namespace) verses upstream package names. In Alpine, a notable example of this would be the lua packages, e.g. `lua5.1`, `lua5.2`, etc. ...Security scanners have issues with disambiguating distribution package names (which exist in a flat namespace) verses upstream package names. In Alpine, a notable example of this would be the lua packages, e.g. `lua5.1`, `lua5.2`, etc. In these cases, scanners are unable to deduce that `lua5.1` is equivalent to upstream `lua~5.1` (e.g. lua 5.1.x).
An emergent industry standard to allow for disambiguation is the [Package URL specification](https://github.com/package-url/purl-spec). We should store this data in APKs, to help the scanners disambiguate.
(This would also require us to add PURLs where needed in our package build recipes, e.g. aports et al. But this is out of scope for here.)https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/577ERROR: Failed to open apk database: UNTRUSTED signature2024-03-07T15:58:11ZIgor ZagorskyERROR: Failed to open apk database: UNTRUSTED signaturecoAlpine-2.0
colin:~\# apk —version
apk-tools 2.0.6-r1
colin:~\# apk update
ERROR: /mnt/cofs0/packages: Bad repository signature
ERROR: Failed to open apk database: UNTRUSTED signature
file /etc/apk/repositories:
`#/media/cdro...coAlpine-2.0
colin:~\# apk —version
apk-tools 2.0.6-r1
colin:~\# apk update
ERROR: /mnt/cofs0/packages: Bad repository signature
ERROR: Failed to open apk database: UNTRUSTED signature
file /etc/apk/repositories:
`#/media/cdrom/apks
/mnt/cofs0/packages
#http://dl-3.alpinelinux.org/alpine/v2.0/packages/main
#http://dl-3.alpinelinux.org/alpine/v2.1/packages/main
http://dl-3.alpinelinux.org/alpine/edge/packages/main
`
What I need to run apk-commands without —allow-untrusted ?
*(from redmine: issue id 577, created on 2011-04-12, closed on 2012-04-26)*https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10892Fuzz apk-tools2024-02-23T12:54:00ZjvoisinFuzz apk-toolsIt would be nice to have some [fuzzing]( https://en.wikipedia.org/wiki/Fuzzing ) for apk-tools, since it's doing some cryptographic operations, has a lot of privileges, parses things coming from the internet, …
I would have done it myse...It would be nice to have some [fuzzing]( https://en.wikipedia.org/wiki/Fuzzing ) for apk-tools, since it's doing some cryptographic operations, has a lot of privileges, parses things coming from the internet, …
I would have done it myself, but since apk is using a database, it's non-trivial to write proper fuzzers :/backloghttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10962Use parallelism optimally2024-02-23T12:34:58ZSander MaijersUse parallelism optimallyThis is a meta-issue for:
- [ ] #5977
- [ ] #10760
- [ ] #10963This is a meta-issue for:
- [ ] #5977
- [ ] #10760
- [ ] #10963https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/5977Parallel Downloads for "apk add"2024-02-23T12:34:41ZNoah NordrumParallel Downloads for "apk add"I think it would be great to have the option to do parallel downloads of
packages, with a configurable amount of threads. Something like “apk
—no-cache —parallel=8 add openjdk8”. (with a default of 1)
You could also grab the various AP...I think it would be great to have the option to do parallel downloads of
packages, with a configurable amount of threads. Something like “apk
—no-cache —parallel=8 add openjdk8”. (with a default of 1)
You could also grab the various APKINDEXes using that same thread pool.
If you didn’t want to hammer the CDNs, you could add a hard-coded
exemption to \*.alpinelinux.org or something.
I’m running a local mirror of dl-cdn.alpinelinux.org, so I would really
like to take advantage of my internal bandwidth.
Since all the dependency information is in APKINDEX, I would think it
wouldn’t be too painful to throw it in a prioritized queue and delegate
to an executor of some kind, but I haven’t looked at the internals
(yet).
*(from redmine: issue id 5977, created on 2016-07-27)*v3.1Timo TeräsTimo Teräshttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10963Parallel installation2024-02-23T12:34:40ZSander MaijersParallel installationSometimes a need for network I/O parallelism, for performance efficiency, can be avoided.
For example, repeated Alpine Linux-based container image builds can use a cache for `/var/cache/apk` across build runs.
However, installing appears...Sometimes a need for network I/O parallelism, for performance efficiency, can be avoided.
For example, repeated Alpine Linux-based container image builds can use a cache for `/var/cache/apk` across build runs.
However, installing appears to happen sequentially.
Since packages should not overlap on the filesystem, and their dependencies/integrity can be calculated, in principle, parallel installation must be possible.
On the other hand, package installation parallelism may have interactions with a transactional or more specifically atomicity-related architecturally significant requirements.https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10887feature to keep old/previous package when upgrading kernel2024-02-23T12:28:43ZNatanael Copafeature to keep old/previous package when upgrading kernelTwo problems:
- when kernel package is upgraded, modprobe will stop working til machine is rebooted, because old kernel modules were deleted.
- it would be nice to keep old kernel around for a while, in case new kernel is broken. For ex...Two problems:
- when kernel package is upgraded, modprobe will stop working til machine is rebooted, because old kernel modules were deleted.
- it would be nice to keep old kernel around for a while, in case new kernel is broken. For example as `/boot/vmlinuz-lts.old`
Possible solution:
We could have a virtual `.old` or `.previous` package. When flagged packages (`linux-lts`would have a flag of some sort) apk would on upgrade transfer ownership of the old package to this virtual package. Files that would be deleted are kept, and files that would be replaced could get a `.old` suffix.
The result would be that `/lib/modules/<old-version>` are transferred to virtual `linux-lts.old`, and `/boot/vmlinuz-lts` are kept as `/boot/vmlinuz-lts.old`. User can then chose to keep this til next time linux-lts is upgraded or for example clean up .old as last step during bootup. (for example `apk del '*.old'` or similar)
This way `modprobe` will continue work after an `apk upgrade` and user can have a vmlinuz-lts.old fallback in boot loader in case new kernel is broken.
Any package could be flagged with 'keep-old' but I think it mostly makes sense for linux kernels.backloghttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10663Remove the need for gzip apk2024-02-19T15:53:42ZFredrik GustafssonRemove the need for gzip apkToday and apk package is built by data.tar.gz, control.tar.gz and signature.tar.gz. Sometime (when using apk in yocto) the speed of build is more important than the size of the packages. Therefore it should be possible to create apk-pack...Today and apk package is built by data.tar.gz, control.tar.gz and signature.tar.gz. Sometime (when using apk in yocto) the speed of build is more important than the size of the packages. Therefore it should be possible to create apk-packages with only data.tar, control.tar and signature.tar. That is no compression. apk should have support for installing apk packages with different compression (including no compression).v3.1https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/5369Add support for LZ42024-02-19T15:48:48ZCarlo LandmeterAdd support for LZ4According to the author on: https://github.com/Cyan4973/lz4 lz4
decompresses almost by a factor 7 compared to zlib while keeping
compression at similar level.
Would this be an possible candidate to replace zlib in apk?
*(from redmine...According to the author on: https://github.com/Cyan4973/lz4 lz4
decompresses almost by a factor 7 compared to zlib while keeping
compression at similar level.
Would this be an possible candidate to replace zlib in apk?
*(from redmine: issue id 5369, created on 2016-04-05)*v3.1Timo TeräsTimo Teräshttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10952apk3 may create entries with 000 permissions2024-02-19T15:10:33ZDaniel Kolesaapk3 may create entries with 000 permissionsThere are some users who have reported to me that apk very rarely creates directories with zero permissions. This is usually not reproducible upon subsequent (re)installation, the packages themselves have correct ones. I have no idea how...There are some users who have reported to me that apk very rarely creates directories with zero permissions. This is usually not reproducible upon subsequent (re)installation, the packages themselves have correct ones. I have no idea how to reproduce this properly.https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10975Remove .keep_* special case2024-02-19T13:53:29ZSertonixRemove .keep_* special caseapk includes a special case when installed directories have a name starting with `.keep_`.
[`src/database.c#L2623-L2624`](https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/c15eb020ffcbdac6a0c658100ec2e41da2d6e15f/src/database.c#L26...apk includes a special case when installed directories have a name starting with `.keep_`.
[`src/database.c#L2623-L2624`](https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/c15eb020ffcbdac6a0c658100ec2e41da2d6e15f/src/database.c#L2623-L2624)
```c
if (bfile.len > 6 && memcmp(bfile.ptr, ".keep_", 6) == 0)
return 0;
```
The code was there since the initial commit. I don't think it provides any purpose.
I would suggest removing that code or documenting it.
CC @fabled since you committed that codehttps://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10848Ignore empty package names2024-02-14T15:36:58ZMarcin KielarIgnore empty package names# Context
We use a sh/bash _trick_ to embed inline comments when we upgrade packages to fix vulnerabilities in our Docker Images. The _trick_ looks like this:
```dockerfile
RUN ... some commands ... \
&& apk add --no-cache --upgrade...# Context
We use a sh/bash _trick_ to embed inline comments when we upgrade packages to fix vulnerabilities in our Docker Images. The _trick_ looks like this:
```dockerfile
RUN ... some commands ... \
&& apk add --no-cache --upgrade \
some-package "$(: 'Fixes CVE-1234')" \
&& ...some more commands...
```
This keeps our Docker Image nice and tidy (just one layer to install all dependencies). We use that with `yum` and `apt-get`, as we have images build from AmazonLinux2 and Ubuntu, and the trick works nicely there.
We also have images built from Alpine, though, which is where the _trick_ doesnt work.
# Example
Consider following `Dockerfile`:
```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:5.0-alpine AS base-build
#hadolint ignore=DL3013,DL3018
RUN apk add --no-cache \
jq \
curl \
python3 \
py3-pip \
&& apk add --no-cache --upgrade \
pcre2>10.40-r0 "$(: 'Fixes https://security.snyk.io/vuln/SNYK-ALPINE315-PCRE2-2869384')" \
&& pip3 install --no-cache-dir --upgrade \
pip \
&& pip3 install --no-cache-dir \
awscli \
&& rm -rf /var/cache/apk/*
```
When building this, the `apk add --no-cache --upgrade pcre2...` command fails. This is because `apk` is given two strings which it interprets as package names:
- the `"pcre2>10.40-r0"` which properly identifies what we want to install
- the `""` which is a result of the _inline comment trick_ we use.
The error message is:
```
#5 1.223 ERROR: unable to select packages:
#5 1.273 (no such package):
#5 1.273 required by: world[]
```
# Feature request
Make `apk` ignore empty-string package names. This way it will work the same way as `apt-get` and `yum`.
# Existing workarounds
Removing the double quotes from the _inline comment trick_ works:
```dockerfile
RUN ... some commands ... \
&& apk add --no-cache --upgrade \
some-package $(: 'Fixes CVE-1234') \
&& ...some more commands...
```
However, this additionally triggers a `hadolint` warning:
```
Dockerfile:4 SC2046 warning: Quote this to prevent word splitting.
```
We can add an ignore for the warning, but we'd much rather have `apk` just ignore empty strings.v3.0https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10944[Feature request] Add ability to disable apk compression2024-02-14T12:41:20Zandrew-aladjev[Feature request] Add ability to disable apk compressionHello everyone, I want to introduce zstd compression in terms of storing multiple packages in binary form.
```sh
mkdir node
cd node
wget https://nodejs.org/download/release/v18.18.0/node-v18.18.0-linux-x64.tar.gz
wget https://nodejs.org...Hello everyone, I want to introduce zstd compression in terms of storing multiple packages in binary form.
```sh
mkdir node
cd node
wget https://nodejs.org/download/release/v18.18.0/node-v18.18.0-linux-x64.tar.gz
wget https://nodejs.org/download/release/v18.17.1/node-v18.17.1-linux-x64.tar.gz
wget https://nodejs.org/download/release/v18.17.0/node-v18.17.0-linux-x64.tar.gz
wget https://nodejs.org/download/release/v18.16.1/node-v18.16.1-linux-x64.tar.gz
wget https://nodejs.org/download/release/v18.16.0/node-v18.16.0-linux-x64.tar.gz
tar -cvf node.tar node-* && zstd -f --ultra -22 node.tar
> ( 169 MiB => 148 MiB, node.tar.zst)
```
Here we have downloaded similar binary node releases. But actually gunzip compression is a problem, zstd can't use binary similarity as it should be.
```sh
gzip -d node-*
gzip -5 node-*
tar -cvf node.tar node-* && zstd -f --ultra -22 node.tar
( 172 MiB => 149 MiB, node.tar.zst)
gzip -d node-*
gzip -1 node-*
tar -cvf node.tar node-* && zstd -f --ultra -22 node.tar
( 196 MiB => 162 MiB, node.tar.zst)
gzip -d node-*
tar -cvf node.tar node-* && zstd -f --ultra -22 node.tar
( 612 MiB => 96.4 MiB, node.tar.zst)
```
Changing gunzip level won't help a lot. But after removing compression zstd finds binary similarity and provided excellent results. Similar things are related to zstd with dictionary: you can train dictionary for similar binary packages and store all package versions very efficiently in zstd archive.
The only one issue is gunzip compression. Can you please add an ability for apk user for remove it? Thank you.https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10971mkpkg -f flag conflicts with global -f, --force flag2024-02-14T12:10:29ZWesley Mooremkpkg -f flag conflicts with global -f, --force flagSimilar to #10822 `mkpkg` has a short flag for `--files` of `-f` but this conflicts with the global `-f, --force` flag. If `-f` is passed then `mkpkg` does not include any files in the package.Similar to #10822 `mkpkg` has a short flag for `--files` of `-f` but this conflicts with the global `-f, --force` flag. If `-f` is passed then `mkpkg` does not include any files in the package.https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/10966apk3: apk fix --no-chown2024-02-05T12:35:47ZSertonixapk3: apk fix --no-chown`apk add` has the `--no-chown` flag. Since `apk fix` (and maybe others) can also trigger file creation it would make sense to add the flag there too.
I think it would fit under the commit options.`apk add` has the `--no-chown` flag. Since `apk fix` (and maybe others) can also trigger file creation it would make sense to add the flag there too.
I think it would fit under the commit options.