News for hackers. Stuff that matters.
Re:An Ubuntu kernel bug causes container crashes by akx on Wednesday July 06, 2022 @02:16AM (#31997371)
> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run.

Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.

Re:An Ubuntu kernel bug causes container crashes by capableweb on Wednesday July 06, 2022 @05:28AM (#31998626)
Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
Re:An Ubuntu kernel bug causes container crashes by dspillett on Wednesday July 06, 2022 @05:42AM (#31998709)
If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu?

Failing that you could try the “single” directive and poke other configurations once booted in that mode.

A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).

Re:An Ubuntu kernel bug causes container crashes by bravetraveler on Wednesday July 06, 2022 @06:46PM (#32007854)
Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem.

There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.

Re:An Ubuntu kernel bug causes container crashes by jacquesm on Wednesday July 06, 2022 @05:38AM (#31998684)
That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
Re:An Ubuntu kernel bug causes container crashes by markstos on Wednesday July 06, 2022 @09:56AM (#32000944)
AWS at least provides serial console access so have the option to access it during the boot cycle.

Alternatively, you umount the drive, attach it to another machine, chroot into it, fix grub or whatever, reverse the process and boot again. It's a few steps, but can be done in a few minutes with practice.

Re:An Ubuntu kernel bug causes container crashes by bravetraveler on Wednesday July 06, 2022 @10:58AM (#32001955)
Out of band management is common and highly recommended
Re:An Ubuntu kernel bug causes container crashes by mrintegrity on Thursday July 07, 2022 @02:10AM (#32010493)
Actually networking and ssh come up for a couple of seconds before containerd triggers the kernel panic so you can fix it by doing this:

while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done

While rebooting the system

Re:An Ubuntu kernel bug causes container crashes by gtirloni on Wednesday July 06, 2022 @08:21AM (#31999773)
Sorry, what? You don't lose complete control of a server because it's rebooting nonstop.