Add support for variables in /etc/apk/repositories like Yum has

This would IMO solve #10756 (closed) by replacing the repo in /etc/apk/repositories by a caching proxy

An example need of such a feature is documented in the wiki: https://wiki.alpinelinux.org/wiki/Alpine_newbie_apk_packages#New_users:_installing_needed_packages

# old /etc/apk/repositories from the example above
cat > /etc/apk/repositories << EOF; $(echo)
http://dl-cdn.alpinelinux.org/alpine/v$(cat /etc/alpine-release | cut -d'.' -f1,2)/main
EOF

would become:

# proposed /etc/apk/repositories
cat > /etc/apk/repositories << EOF; $(echo)
http://dl-cdn.alpinelinux.org/alpine/v\$releasever/main
EOF

Now that is of little use, BUT we use Artifactory as cache and thus point our Alpine containers to a repo mirror there... but need a separate repo file for every alpine release or synthesize a repo file based on the alpine release... that sounds easy enough but the code that needs to synthesize the repo file needs to be running outside the container and thus has to guess the correct release number from docker tags for various base images (node, go, java, ...)...

We would love to be able to just bind mount /etc/apk/repositories with our artifactory repositories (and credentials) without fancy magic and use $releasever as a placeholder for the version inside them.

Running into this same issue at this very moment. We also use Artifactory as both a pass-through cache as well as for hosting internal packages.

mentioned in issue #10787 (closed)

changed milestone to %backlog

The ground work for this has been done, and it would be simple to add support for this now.

It would be matter of determining which variables apk should substitute when parsing repositories.

I don't want to support any distro specific files such as /etc/alpine-release as apk is now used in various other distros.

So I wonder which substitions would be needed? Where to read the variable values from? I suppose general usage would be ALPINE_STABLE and perhaps ALPINE_MIRROR. In which case both could be setup in etc/apk/vars or similar configuration file, or perhaps just global options that can be read from the global config?

An interesting idea would be that the new mkndx could contain information that a new stable release is out and include value for ALPINE_STABLE to update to it. A new option upgrade --release or similar could then just automatically use the index variable to automatically override the variable and ugprade things. In this case the etc/apk/vars would need to be packaged be upgrade on distro ugprade, or apk would need to update it after the distro upgrade.

@ncopa @ariadne @aparcar @q66 @sertonix Thoughts or comments?

There is also /etc/os-release which is sort of a standard.

There is also /etc/os-release which is sort of a standard.

Yes. Initially I thought it does not contain the needed info since VERSION_ID is the full version tag. But I suppose we could add some sort of string manipulation to translate that to the version. If using this, ideally there would be some variable that equals the repository branch component (v3.21 / edge).

Probably something additional would be needed for the mirror selection.

I am uncertain if reading /etc/os-release (or similar) can be used reasonably. For example if one uses the VERSION_ID= variable they would need to change /etc/os-release to upgrade to the next major version? The file isn't supposed to be edited by users though.

I am uncertain if reading /etc/os-release (or similar) can be used reasonably. For example if one uses the VERSION_ID= variable they would need to change /etc/os-release to upgrade to the next major version? The file isn't supposed to be edited by users though.

So it would work if os-release is read for the defaults. User could still override via command line.

Thinking more, index should not provide overrides for these variables as suggested by my earlier comment. Instead some base package could provide a file like etc/apk/next-release or similar, that contain the variable overrides to be used with apk upgrade --next-release.

Instead some base package could provide a file like etc/apk/next-release or similar, that contain the variable overrides to be used with apk upgrade --next-release.

Not a huge fan of a special --next-release option but I would be ok with that. We would need to store the current release in a file (etc/apk/release comes to my mind but the name might be confusing since it sounds like the version of apk). Also we would need to make sure that --next-release doesn't cause problems when used on alpine edge and other rolling releases. And --next-release should probably imply --update-cache (ideally with #11061 (closed) fixed).

A generic question with variables is if the cache name should consider the values of the variables or not (both have pros and cons).

I would very much like this, as it would allow me to implement good support for mirror selection with the way repository files are currently installed in Chimera.

I don't really care much where it's read from, as long as the following is met:

1. I can specify fallback variables (e.g. default mirror base URL) and these live in some file that is not in /etc.
2. Then, people can override these defaults, using some file in /etc.
3. Perhaps, multiple files should be read? E.g. some kind of vars.d thing (maybe in both locations), not sure though.

Btw, speaking of os-release, the recommended path by the specification these days is /usr/lib/os-release. The /etc/os-release remains as a symlink.

So I'm thinking these variable should apply only to repositories file handling. And put them directly into the repositry.list files with some !define type keywords, or perhaps just in the same repositories.d directories as *.vars.

mentioned in issue #11069 (closed)

And put them directly into the repositry.list files with some !define type keywords, or perhaps just in the same repositories.d directories as *.vars.

I would prefer if they are included in repositry.list. I was thinking of just <key>=<val> if we restrict the characters that can be used in <key> (alphanumeric with _-?) this should be easy to distinguish from lines beginning with / or {file,https,http}://.

I was thinking of just <key>=<val> if we restrict the characters that can be used in <key>

There was also request for potentially other directives in the .list files, e.g. to set the default repository line format !default [v2|v3] or similar. So I would probably like have some sort of keyword syntax rather then just plain assignment.

Also, this means the processing of the .list files need to be well defined. Because the variable value can depend on processing stage if its defined in multiple files.

So probably it would need to be something like:

1. resolve the path for each configuration file name to load (assuming #11069 (closed) is implemented)
2. sort all filenames (without directory portion)
3. pass 1, read all files and process the variable assignments etc
4. pass 2, expand repository lines to index download locations

This would allow basically allow /lib/apk/repositories.d/0-distro.list to define repository lines and distro_mirror and distro_compoments with their defaults. And user can then override those variables in a new file in /etc/apk/repositories.d/my-config.list or similar -- or completely disable the defaults by creating an empty /etc/apk/repositories.d/0-distro.list.

The above should cover things, and perhaps the only confusion for user might be if same variable is defined in several places to determine which one is actually used.

As alternative, Yum has a directory where it's just file per variable /etc/yum/vars/$varname and the contents will be the variable value. I suppose this leaves less confusion for the declaration location. But is a bit too much of files for my taste, so I'm leaning towards keeping things in the .list files.

mentioned in commit 5c98cf56

mentioned in commit 8b45b196

mentioned in commit 47842ec6

So the MR is now done mostly, and supports !set key=val syntax. I'm thinking still if it should be .set key=val or just plain set key=val instead to make it nicer to read. Though, to keep things simple, the problem is that the parser needs to be able to distinguish if the first word is a keyword or local relative path (used in --repository option). Technically a .something is more likely to be a path than !something.

Of course we could enforce that count as url/path the url must start with scheme (contain ://, or start with / or ./), but this would be change to the format. Or just reject relative paths from the .list files and disable keyword parsing from command line options. Not sure if it makes sense to support --repository "!set val=foo".

Also need to document and check that the key is limited to specific characters (e.g. no spaces/control chars).

mentioned in commit 21f19fcd

mentioned in commit 2e88391e

mentioned in commit 9ece632b

mentioned in commit f84cf644

mentioned in commit 630ff194

mentioned in commit 460d62ee

closed with merge request !266 (merged)