alpine-ec2-ami merge requestshttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests2019-10-26T20:28:13Zhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/55fix regression on nvme mdev script2019-10-26T20:28:13ZJake Buchholz Göktürkfix regression on nvme mdev script* `sh` doesn't allow nesting of prefix strip
* also update some minor test profile bits
* built new AMI revisions with fixed nvme* `sh` doesn't allow nesting of prefix strip
* also update some minor test profile bits
* built new AMI revisions with fixed nvmehttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/56alpine 3.10.2 (and latest edge) AMIs2019-08-26T03:51:31ZJake Buchholz Göktürkalpine 3.10.2 (and latest edge) AMIshttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/58Alpine-3.10.3 (and latest edge)2019-12-20T03:32:49ZJake Buchholz GöktürkAlpine-3.10.3 (and latest edge)Also includes fix for Packer regression affecting encrypted AMIsAlso includes fix for Packer regression affecting encrypted AMIshttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/60Alpine 3.11.02019-12-20T04:27:38ZJake Buchholz GöktürkAlpine 3.11.0Also attach `revision` per build target -- i.e. 3.11.0 is r0 (the default), but 3.10.3 remains r1.Also attach `revision` per build target -- i.e. 3.11.0 is r0 (the default), but 3.10.3 remains r1.https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/61Alpine 3.11.22020-01-04T20:24:22ZJake Buchholz GöktürkAlpine 3.11.2https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/62The Latest Releases2020-02-06T04:29:44ZJake Buchholz GöktürkThe Latest Releases* 3.9.5
* 3.10.4
* 3.11.3* 3.9.5
* 3.10.4
* 3.11.3https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/63Alpine 3.11.52020-04-26T00:03:42ZJake Buchholz GöktürkAlpine 3.11.5https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/653.11.6, 3.10.5, and 3.9.62020-05-30T22:53:49ZJake Buchholz Göktürk3.11.6, 3.10.5, and 3.9.6https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/66Tooling updates2020-05-30T22:10:09ZMike Crutemike@crute.usTooling updatesThis is kind of a big one.
This change combines all of the various python scripts into one builder script and uses argparse subcommands to handle dispatch to the other commands. It also removes a lot of hand-rolled command line manipu...This is kind of a big one.
This change combines all of the various python scripts into one builder script and uses argparse subcommands to handle dispatch to the other commands. It also removes a lot of hand-rolled command line manipulation in favor of using argparse. It also tries to be resilient to path locations but always finding the git root before operating on paths. The rationale behind the major refactor is that we need to share logic between a bunch of the scripts now and that's easier to do if they're all in one file. That also seemed easier than making an importable python library for less than 700 lines of code.
The `make-amis` script was converted from shell to python. In the process identity broker integration was added so authentication happens in the region the build is happening, which is passed through to packer. This should eliminate needing to export credentials from the identity broker directly. Building will still require exporting `IDENTITY_BROKER_API_KEY` (which is available on the identity broker home page). In the future I hope to be able to use GitHub actions to do the builds and releases so we don't have to run them by hand, this will be a lot easier with the new python implementation.
There's a new identity broker client that can handle multi-regional credentials and provides a list of all regions, including the ability to get a credential for regions that are opted-in for an account. Any region iteration logic should use the broker going forwards instead of the EC2 `describe_regions` function from boto3 (the broker uses similar logic behind the scenes but with more checks).
The runtime services (`svcs`) was remodelled to use a nested map in the profiles. I think this is a little easier to read through at a glance and it greatly simplified the transform of that structure into a flat list.
The `resolve-profiles` script was very heavily modified. The goal was to make the logic as declarative as possible and eliminate some of the unused code there. I think that was accomplished and it made adding some new key transforms pretty easy.
Additionally I removed all region manipulation logic. There will be a PR forthcoming after the ARM release that adds a release tool that handles AMI replication and permission updates to make them public. This will also handle the opt-in regions case. I'll link that work to #53
The rest of the scripts remain largely unchanged, just shuffled around to fit into classes.
The final major change I hope to make to this tooling is an update to the `prune-amis` subcommand to resolve #23 but we can talk about that over there.Mike Crutemike@crute.usMike Crutemike@crute.ushttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/69Optional AWS Profile & Cross-Account Access2020-08-09T22:06:56ZJake Buchholz GöktürkOptional AWS Profile & Cross-Account AccessAllows encoding of the AWS profile to use in the build profile, and enabling the built AMI with a list of AWS accounts that are allowed access.Allows encoding of the AWS profile to use in the build profile, and enabling the built AMI with a list of AWS accounts that are allowed access.https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/78Fix edge builds2020-08-09T22:11:36ZJake Buchholz GöktürkFix edge buildsalpine-mirrors package removed after 3.12
also add profile for 3.12
Resolves #77 alpine-mirrors package removed after 3.12
also add profile for 3.12
Resolves #77 https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/79Optional Additional Setup2020-08-13T01:12:24ZJake Buchholz GöktürkOptional Additional SetupProfiles can specify `setup_script` to do additional things. If additional files/dirs are required, a `setup_copy` map will copy them to the build instance so that `setup_script` can use/install them.
Also implements `ami_user` allow...Profiles can specify `setup_script` to do additional things. If additional files/dirs are required, a `setup_copy` map will copy them to the build instance so that `setup_script` can use/install them.
Also implements `ami_user` allowing profiles to change the AMI user from the default, "alpine".https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/80Fix 'revision' and 'end_of_life'2020-08-17T18:11:52ZJake Buchholz GöktürkFix 'revision' and 'end_of_life'Also...
* update alpine.conf with 3.12
* update apk-tools and alpine-keys
* use test profile to test fixes and newer features
Resolves #75 Also...
* update alpine.conf with 3.12
* update apk-tools and alpine-keys
* use test profile to test fixes and newer features
Resolves #75 https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/81Fix nvme-ebs-links Installation2020-08-18T02:06:10ZJake Buchholz GöktürkFix nvme-ebs-links Installationensure that ownership and permissions are set properlyensure that ownership and permissions are set properlyhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/83New Release Tool2020-12-12T02:02:15ZMike Crutemike@crute.usNew Release ToolHere's the new release tool. It uses the identity broker to acquire credentials for all activated regions and copy the AMI, including tags, to that region. For future accounting it also adds a `source_ami` tag. Everything should also wor...Here's the new release tool. It uses the identity broker to acquire credentials for all activated regions and copy the AMI, including tags, to that region. For future accounting it also adds a `source_ami` tag. Everything should also work if the user isn't using the identity broker, so long as AWS credentials are accessible by the SDK and `--region` is passed (one or more times) to specify target regions. The tool will try not to copy the AMI more than once to a region, using the `source_ami` tag, so if it's run multiple times for the same source AMI it will copy to new regions and fix permissions if needed.
The flip to public permissions is designed to happen all at once, post-copy, in the linear flow so that a release looks like it happens approximate at the same time.
I've tested the identity broker path as well as the permissions fix path but have not tested the standalone path since that isn't a use-case I have and I don't have a test account handy right now; any testing there would be appreciated. I also haven't tested sharing with separate accounts `--allow-accounts` instead of `--public`, but that should work as well.
To use the identity broker, grab the API key from the broker homepage and export it as the environment variable `IDENTITY_BROKER_API_KEY`. Everything else should just work from there. The token is valid for 6 hours. Note that there are pretty aggressive rate limits on the broker for getting credentials so if you're doing a lot of testing in a row you'll end up waiting for the timeouts, but the script should handle it gracefully.
Any feedback would be appreciated. This should unblock the 3.12 release. There's more stuff coming with tools to prune AMIs and build the catalog from the tag metadata instead of YAML files; but I'll follow up with those a little later.Jake Buchholz GöktürkJake Buchholz Göktürkhttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/84udhcpc hooks for ENI IPv6 & secondary IPv42020-09-03T22:04:31ZJake Buchholz Göktürkudhcpc hooks for ENI IPv6 & secondary IPv4Automatically sets up any IPv6 and secondary IPv4 on instance ENIs when DHCP leases are bound or renewed on that interface.
Resolves #70
Freshly instantiated instance...
```
~ % ssh -i AWS/tomalok.pem alpine@2600:1f13:224:d501:c9...Automatically sets up any IPv6 and secondary IPv4 on instance ENIs when DHCP leases are bound or renewed on that interface.
Resolves #70
Freshly instantiated instance...
```
~ % ssh -i AWS/tomalok.pem alpine@2600:1f13:224:d501:c953:a14a:7b8:3909
Warning: Permanently added '2600:1f13:224:d501:c953:a14a:7b8:3909' (ECDSA) to the list of known hosts.
Welcome to Alpine!
The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.
You can setup the system with the command: setup-alpine
You may change this message by editing /etc/motd.
alpine@ip-172-30-30-85:~$ cat /var/log/messages
Aug 30 02:00:58 localhost syslog.info syslogd started: BusyBox v1.32.0
Aug 30 02:00:58 localhost daemon.info init: starting pid 2346, tty '': '/sbin/openrc default'
Aug 30 02:00:59 localhost daemon.notice udhcpc/post-bound[2406]: eth0 add 2600:1f13:224:d501:c953:a14a:7b8:3909 - success
Aug 30 02:01:00 localhost daemon.info chronyd[2517]: chronyd version 3.5.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP -SCFILTER +SIGND +ASYNCDNS -SECHASH +IPV6 -DEBUG)
Aug 30 02:01:00 localhost daemon.warn chronyd[2517]: Could not read valid frequency and skew from driftfile /var/lib/chrony/chrony.drift
Aug 30 02:01:05 localhost daemon.info chronyd[2517]: System's initial offset : 0.428769 seconds slow of true (slew)
Aug 30 02:01:07 localhost auth.info sshd[2551]: Server listening on 0.0.0.0 port 22.
Aug 30 02:01:07 localhost auth.info sshd[2551]: Server listening on :: port 22.
Aug 30 02:01:07 localhost user.info : password for 'root' changed
Aug 30 02:01:07 localhost user.info : password for 'alpine' changed
Aug 30 02:01:08 localhost daemon.info init: starting pid 2632, tty '/dev/ttyS0': '/sbin/getty -L ttyS0 115200 vt100'
Aug 30 02:01:11 localhost daemon.info chronyd[2517]: Selected source 169.254.169.123
Aug 30 02:03:32 localhost auth.info sshd[2638]: Accepted publickey for alpine from 2601:602:8f80:75b9:a567:6cb7:c10e:41bb port 57388 ssh2: RSA SHA256:NXpuX150N2HDjZ8bBv9UnjC1gi52KLrvtSyLiHqW8FA
alpine@ip-172-30-30-85:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:5e:2d:89:26:ed brd ff:ff:ff:ff:ff:ff
inet 172.30.30.85/20 brd 172.30.31.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 2600:1f13:224:d501:c953:a14a:7b8:3909/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::5e:2dff:fe89:26ed/64 scope link
valid_lft forever preferred_lft forever
```Mike Crutemike@crute.usMike Crutemike@crute.ushttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/85ENI Hotplugging, etc.2020-09-16T05:17:18ZJake Buchholz GöktürkENI Hotplugging, etc.ENI Hotplug / udhcpc script
* works with all Alpine versions back to 3.9
* udhcpc handles ENI's primary IPv4
* post-bound/post-renews eth-eni-hook handles secondary IPv4 & IPv6 addresses, route tables, and rules
setup-ami tweaks
*...ENI Hotplug / udhcpc script
* works with all Alpine versions back to 3.9
* udhcpc handles ENI's primary IPv4
* post-bound/post-renews eth-eni-hook handles secondary IPv4 & IPv6 addresses, route tables, and rules
setup-ami tweaks
* move scripts to be installed into setup-ami.d/
* move config snippets into setup-ami.d/etc/ (previously embedded in setup-ami)
resolves #82, #70 https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/87eth-eni-setup init script2020-09-22T02:43:35ZJake Buchholz Göktürketh-eni-setup init scriptbefore networking starts up, makes sure eth interfaces match attached ENIs
also fixes a permissions problem with eth-eni-hotplug mdev config
resolves #86 before networking starts up, makes sure eth interfaces match attached ENIs
also fixes a permissions problem with eth-eni-hotplug mdev config
resolves #86 https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/88Update to 3.12.12020-10-30T18:34:50ZJake Buchholz GöktürkUpdate to 3.12.1* Update to Alpine Linux v3.12.1
* Switch to using t3a instances for x86_64 builds* Update to Alpine Linux v3.12.1
* Switch to using t3a instances for x86_64 buildshttps://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami/-/merge_requests/89Update Releases for 3.12.12020-11-05T16:17:36ZJake Buchholz GöktürkUpdate Releases for 3.12.1* AMIs for Alpine release **3.12.1** & today's **edge**
* Introducing **aarch64** AMIs for ARM-based EC2 instances
* Supports **IMDSv2**, **hotplug ENIs**, **IPv6** and **secondary IPv4** addresses
* New AMIs available in 20-ish regi...* AMIs for Alpine release **3.12.1** & today's **edge**
* Introducing **aarch64** AMIs for ARM-based EC2 instances
* Supports **IMDSv2**, **hotplug ENIs**, **IPv6** and **secondary IPv4** addresses
* New AMIs available in 20-ish regions!