VMs end up locked out on Digital Ocean
I have tried the following disk images on Digital Ocean and found they do not work:
- https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/cloud/nocloud_alpine-3.20.3-x86_64-bios-tiny-r0.qcow2
- https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/cloud/nocloud_alpine-3.20.3-x86_64-bios-cloudinit-r0.qcow2
- https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/cloud/generic_alpine-3.20.3-x86_64-bios-cloudinit-r0.qcow2
Specifically, in each case the VM starts and ping works, but the VM does not allow to log in over SSH using the pre-configured public SSH key: neither root
not alpine
work. With VMs installed from Digital Ocean selection of Linux distributions this works. On the VM web/VGA console there are only the following lines:
Loading vmlinuz-virt... ok
Loading initramfs-virt...ok
Pressing Enter does not return a login prompt. The only potential way to recover the VM is to boot from a virtual rescue CD (not tried).
Following this experience, I tried the same disk images in a local VirtualBox and observed exactly the same [lack of] output on the VGA console. However, I was able to recover the VM and gather some diagnostics as to why the problem occurs and how it could be fixed. Let me state some facts first:
- DO VMs use SeaBIOS and start booting from virtual disk after a very short delay.
- DO web/VGA console takes several seconds to attach to a VM, disconnects during the VM power-cycle and does not allow to send Ctrl-Alt-Del.
- After the VM has booted as described above, the only way DO provides to restart it is power-cycling.
- In the above images syslinux boot timeout is 1 second.
- In the above images the only console parameter in the default kernel command line is
console=ttyS0,115200n8
. - DO does not implement serial port console for VMs.
- In the above images
/etc/inittab
includes the following lines, which do not change automatically, although another Linux user points out this code looks like it is supposed to uncomment them conditionally:#tty1::respawn:/sbin/getty 38400 tty1 #tty2::respawn:/sbin/getty 38400 tty2 #tty3::respawn:/sbin/getty 38400 tty3 #tty4::respawn:/sbin/getty 38400 tty4 #tty5::respawn:/sbin/getty 38400 tty5 #tty6::respawn:/sbin/getty 38400 tty6
- In a VM installed from the "standard" ISO image the above lines are not commented and VGA console is perfectly usable from the very beginning.
- In VirtualBox it is possible to see the VGA console before syslinux starts, to interrupt the boot timeout, to change kernel command line and to access the virtual serial port; this is more than sufficient to recover the VM without using a 3rd-party rescue boot.
This allows to state the following problems with the above disk images:
- Using DO, it is impossible to change boot parameters. By the time a DO user can access the VGA console, syslinux boot timeout has long expired and the boot has commenced with the default kernel arguments. Suggested fix: raise the timeout to 10 seconds.
- It is impossible to log in from the VGA console after boot, no matter how hosted. Suggested fix: do not disable standard getty by default.
- It is impossible to monitor/debug boot progress on the VGA console, no matter how hosted. Suggested fix: use
console=tty0 console=ttyS0,115200n8
to send kernel boot messages to both VGA and serial port. - Neither cloud-init nor Tiny Cloud work with DO infrastructure, another Linux user points out this is likely because DO expects specific arrangements in the VM configuration.
Please review.