alpine-ec2-ami merge requestshttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests2021-12-26T23:21:30Zhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/118Add support for cloud-init2021-12-26T23:21:30ZMike Crutemike@crute.usAdd support for cloud-initThis pull request adds cloud-init support to the AMI. Upstream cloud-init still has some bugs with their default configuration that I need to patch, mostly around not locking the `alpine` user at init time. But this is a start. Currently...This pull request adds cloud-init support to the AMI. Upstream cloud-init still has some bugs with their default configuration that I need to patch, mostly around not locking the `alpine` user at init time. But this is a start. Currently you can build the image and it'll deploy the SSH keys correctly to the `alpine` user, it will also run the user-data. I haven't done any other quality checking.
There aren't any AMIs available yet, I've deleted the testing ones, but they can be produce by building the `ci-v3_13-x86_64` build from the `alpine` profile. This will generate an AMI with a `alpine-ci-test-ami-` prefix that is bootstrapped using cloud-init. All of the default builds still use `tiny-ec2-bootstrap`.
This change should be safe to merge as-is because it doesn't change the default behavior of the normal targets.https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/128re-add 'sudo' to 3.15, add deprecation note to /etc/motd2021-11-28T00:15:54ZJake Buchholz Göktürkre-add 'sudo' to 3.15, add deprecation note to /etc/motdnew 3.15.0-r1 revision from alpine-cloud-builder
* adds `sudo` back
* adds deprecation note to /etc/motdnew 3.15.0-r1 revision from alpine-cloud-builder
* adds `sudo` back
* adds deprecation note to /etc/motdhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/127fix release YAML: version -> release & 3.15.0 EOL2021-11-24T19:27:29ZJake Buchholz Göktürkfix release YAML: version -> release & 3.15.0 EOLhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/126add missing level in releases YAML2021-11-24T17:56:24ZJake Buchholz Göktürkadd missing level in releases YAMLhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/125Update Releases YAML for 3.15.02021-11-24T17:42:47ZJake Buchholz GöktürkUpdate Releases YAML for 3.15.03.15.0 (etc.) releases have been built/imported/published with alpine-cloud-images builder3.15.0 (etc.) releases have been built/imported/published with alpine-cloud-images builderhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/124Release 3.13.6, 3.12.8, and 3.11.122021-09-01T14:12:09ZJake Buchholz GöktürkRelease 3.13.6, 3.12.8, and 3.11.12https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/123Release 3.14.22021-08-28T04:27:08ZJake Buchholz GöktürkRelease 3.14.2* release 3.14.2
* edge AMIs now use 'doas' instead of 'sudo'
resolves #125* release 3.14.2
* edge AMIs now use 'doas' instead of 'sudo'
resolves #125https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/122Update for Version 3.142021-06-15T19:34:39ZJake Buchholz GöktürkUpdate for Version 3.14* getting ready for 3.14
* remove 3.10 build
* fix aarch64 AMI's /etc/default/grub (resolves #121)
* bump revision for aarch64 3.13 & 3.12 (to rebuild with fix)* getting ready for 3.14
* remove 3.10 build
* fix aarch64 AMI's /etc/default/grub (resolves #121)
* bump revision for aarch64 3.13 & 3.12 (to rebuild with fix)https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/119Remove backport hack for tiny-ec2-bootstrap2021-05-03T16:49:16ZMike Crutemike@crute.usRemove backport hack for tiny-ec2-bootstrapAriadne and I have [backported](https://security.alpinelinux.org/srcpkg/tiny-ec2-bootstrap) tiny-ec2-bootstrap to all supported versions of Alpine so there should be no need for the hack to backport the package any longer.Ariadne and I have [backported](https://security.alpinelinux.org/srcpkg/tiny-ec2-bootstrap) tiny-ec2-bootstrap to all supported versions of Alpine so there should be no need for the hack to backport the package any longer.https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/117new Alpine releases (3.13.5, 3.12.7, 3.11.11, 3.10.9)2021-04-15T01:48:03ZJake Buchholz Göktürknew Alpine releases (3.13.5, 3.12.7, 3.11.11, 3.10.9)https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/116New releases (3.13.4, 3.12.6, 3.11.10, 3.10.8)2021-04-01T01:53:54ZJake Buchholz GöktürkNew releases (3.13.4, 3.12.6, 3.11.10, 3.10.8)Addresses CVE-2021-28831Addresses CVE-2021-28831https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/115Alpine 3.13.2 released2021-02-17T16:49:14ZJake Buchholz GöktürkAlpine 3.13.2 releasedhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/113Autodetect Current Revision of Alpine Version2021-02-03T04:13:35ZJake Buchholz GöktürkAutodetect Current Revision of Alpine Version* continue to use provided 'release' value if specified
* continue to use 'edge' for edge versions
* deduce 'release' value from the version on the alpine-base APK in https:\/\/dl-cdn.alpinelinux.org/alpine/v\<version\>/main/\<arch\>/ ...* continue to use provided 'release' value if specified
* continue to use 'edge' for edge versions
* deduce 'release' value from the version on the alpine-base APK in https:\/\/dl-cdn.alpinelinux.org/alpine/v\<version\>/main/\<arch\>/
* update test profile with 3.13
Resolves #112 https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/111Alpine Linux 3.13.1 released2021-01-30T03:25:22ZJake Buchholz GöktürkAlpine Linux 3.13.1 releasedResolves #110 Resolves #110 https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/108Release Alpine 3.13.02021-01-15T07:53:42ZJake Buchholz GöktürkRelease Alpine 3.13.0https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/106Builder Overhaul2021-01-05T01:36:16ZJake Buchholz GöktürkBuilder OverhaulSubcommands
* merge 'resolve-profiles' and 'make-amis' into 'amis'
* rename 'update-releases' to 'release-yaml'
* rename 'gen-release-readme' to 'release-readme'
* rename 'prune-amis' to 'prune'
* reorder to match the usual workflow...Subcommands
* merge 'resolve-profiles' and 'make-amis' into 'amis'
* rename 'update-releases' to 'release-yaml'
* rename 'gen-release-readme' to 'release-readme'
* rename 'prune-amis' to 'prune'
* reorder to match the usual workflow
* use argparse mutually-exclusive group where appropriate
* use argparse 'metavar' and 'nargs' for more salient help
ReleaseAMIs
* can now specify multiple AMIs on command line
* add explicit '--private' argument
* if no '--private', '--public', or '--allow-account' is specified, default to propagate the source AMI's permissions to its copies
* move 'iter_regions' and 'get*image' methods out of ReleaseAMIs class because they're also used elsewhere
* 'update_image_permissions' resets perms before adding new perms
* pending_copy loop, reports on everything in progress, waits 3m before reporting on everything again, and then waits 30s between reports
* pending_copy also notes when a copy has completed (and only queues for pending_perms if they need adjustment)
Releases class
* used by release-yaml and prune subcommands
* caches region client objects for later use (by prune)
* loads images from region - either from a profile or "unknown" (no profile tag)
* builds the releases object - now structured release -> build (instead of build -> release)
ReleasesReadme
* works with new releases object format
* improve sorting and selection of latest per version per-build AMIs
* empty cell if a region doesn't happen to have a build AMI there
PruneAMIs
* rename 'version' level to 'end-of-life'
* add 'UNKNOWN' pruning level
* works, even if you don't want to --use-broker
* --keep N - keeps an additional N AMIs that would otherwise have been purged per build
* --defer-eol DAYS - give EOL AMIs a grace period past their official EOL date
* no AMI deletion happens unless --no-pretend arg is provided
* improve pruning criteria scan and candidate selection
resolves #102
Also includes updated **releases/README.md** and **releases/alpine.yaml**.https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/104Post-Build Cleanup, etc.2020-12-22T23:42:22ZJake Buchholz GöktürkPost-Build Cleanup, etc.**scripts/builder.py**...
GenReleaseReadme:
* combine with ReleaseReadmeUpdater
* generates `README_<profile>.md`
* `README_alpine.md` is a symlink to `README.md`
* don't crash when README doesn't preexist
* append image list to ...**scripts/builder.py**...
GenReleaseReadme:
* combine with ReleaseReadmeUpdater
* generates `README_<profile>.md`
* `README_alpine.md` is a symlink to `README.md`
* don't crash when README doesn't preexist
* append image list to README if no list found to replace
MakeAMIs:
* collect all artifact IDs and report after all builds
* don't update releases/readme
PruneAMIs:
* defaults to pretend mode, unless `--no-pretend`
* improve readability
UpdateReleases:
* replace code with what was RefreshReleaseshttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/101Release the latest versions2020-12-19T19:16:49ZJake Buchholz GöktürkRelease the latest versions* v3.12.3 (x86_64 & aarch64)
* v3.11.7 (x86_64)
* v3.10.5-r1 (x86_64)
* today's edge (x86_64 & aarch64)
builder.py timings, roughly
* amis - 23m
* release (serial) - 1h38m
* refresh-releases - 4m
* gen-release-readme - instanta...* v3.12.3 (x86_64 & aarch64)
* v3.11.7 (x86_64)
* v3.10.5-r1 (x86_64)
* today's edge (x86_64 & aarch64)
builder.py timings, roughly
* amis - 23m
* release (serial) - 1h38m
* refresh-releases - 4m
* gen-release-readme - instantaneoushttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/97Add refresh-releases subcommand, etc.2020-12-15T06:24:31ZJake Buchholz GöktürkAdd refresh-releases subcommand, etc.* **builder.py**
+ gen-release-readme
- convert `build_time` to int
+ release
- add `source_region` to copied AMI tags
- check source AMI's permissions, queue for fixing, if necessary
+ refresh-releases
- upd...* **builder.py**
+ gen-release-readme
- convert `build_time` to int
+ release
- add `source_region` to copied AMI tags
- check source AMI's permissions, queue for fixing, if necessary
+ refresh-releases
- update releases/<profile>.yaml based on AMIs that exist in regions
* Release Alpine 3.12.2 & today's edge
Resolves #96, related to #95https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/83New Release Tool2020-12-12T02:02:15ZMike Crutemike@crute.usNew Release ToolHere's the new release tool. It uses the identity broker to acquire credentials for all activated regions and copy the AMI, including tags, to that region. For future accounting it also adds a `source_ami` tag. Everything should also wor...Here's the new release tool. It uses the identity broker to acquire credentials for all activated regions and copy the AMI, including tags, to that region. For future accounting it also adds a `source_ami` tag. Everything should also work if the user isn't using the identity broker, so long as AWS credentials are accessible by the SDK and `--region` is passed (one or more times) to specify target regions. The tool will try not to copy the AMI more than once to a region, using the `source_ami` tag, so if it's run multiple times for the same source AMI it will copy to new regions and fix permissions if needed.
The flip to public permissions is designed to happen all at once, post-copy, in the linear flow so that a release looks like it happens approximate at the same time.
I've tested the identity broker path as well as the permissions fix path but have not tested the standalone path since that isn't a use-case I have and I don't have a test account handy right now; any testing there would be appreciated. I also haven't tested sharing with separate accounts `--allow-accounts` instead of `--public`, but that should work as well.
To use the identity broker, grab the API key from the broker homepage and export it as the environment variable `IDENTITY_BROKER_API_KEY`. Everything else should just work from there. The token is valid for 6 hours. Note that there are pretty aggressive rate limits on the broker for getting credentials so if you're doing a lot of testing in a row you'll end up waiting for the timeouts, but the script should handle it gracefully.
Any feedback would be appreciated. This should unblock the 3.12 release. There's more stuff coming with tools to prune AMIs and build the catalog from the tag metadata instead of YAML files; but I'll follow up with those a little later.Jake Buchholz GöktürkJake Buchholz Göktürk