If you have unattended-upgrade and automatic reboot in the cloud to benefit from security updates for long-lived instances, then you better make sure to have a tty console attached to it. You are treating it like a physical machine, you must have the same tooling around.

Not really, console access through IPMI found on most servers

Exceptions tend to be white boxes built with desktop components, at which point, yea. The proverbial You asked for this problem

Not necessarily. With good timing and some luck, you can connect the serial/"recovery" console before GRUB's timeout ends and either change the running kernel or add the `systemd.mask=docker.service` boot parameter to prevent Docker from starting.

Sounds like a VM and not a physical server.

Nope. Back before VMs were thing it was common to do "lights out" style remote management via a console server. That console server would then have a serial connection (the old 9 pin d-sub plug[1]) to your individual physical servers. You could then connect to your remote servers local TTY via the console server a little like jumping to remote servers via an SSH bastion. However it did sometimes require a little bit of prior configuration, depending on your distro[2].

This wasn't just limited to Linux either. It was a common UNIX trick :)

This is a bit of a lost art these days though. iLo, IPMI have replaced the need for serial. Then virtualisation and, to a lesser extent, containerisation have lowered the bar even further plus also moving the industry towards more ephemeral systems that can be destroyed and rebuilt automatically rather than the old habits of nursing failed hosts back to health.

[1] https://duckduckgo.com/?q=9+pin+d-sub+plug&t=newext&atb=v316...

[2] https://www.kernel.org/doc/html/v5.3/admin-guide/serial-cons... (a lot of distros at the time did ship a kernel with this support compiled in. I don't know how common it is now).

And quite a few implementations actually emulate the serial console allowing for the exact same access. (Serial Over Lan or SOL for short.)

Still common on network devices (Cisco, Juniper, Arista etc.). No IPMI or similar on those.

Console servers from the likes of OpenGear and Lantronix still heavily used for those.

Sure. For a physical server, you'd use its lights-out management to the same effect.

If its in the cloud you'd have a virtual console.

Unsurprisingly AWS is ghetto about this.

Or a real server with Lights Out Management.

That’s why “usually” is in the sentence. :)

Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.

I'd be more inclined to say that physical servers usually have some sort of console access available.

I'm not sure I've ever worked with any (2008-present) that don't in any case.

That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server.

Oh well, I guess experiences differ.

My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?

Serial connections will only cost you a Raspberry Pi (there's probably some really cheap console servers on eBay too).

I don't think the issue is so much cost but more this kind of systems administration is becoming a forgotten art because 99% of the time modern tooling removes the need for it. So younger sysadmins are never taught how to do these kinds things. However when I started out, I worked in a few small companies that had their physical hosts connected to a console server (which was a Cisco device like a network switch) via serial cables and you'd then connect to that console server remotely.

Depends on the infra and how it’s set up.

If you can afford to have something down for an extended period then fine. But even with a small team some services are built such that certain device outages cannot be tolerated, at least for an extended period.

So out-of-band/console servers or whatever still make a lot of sense and a relatively high priority.

You can do this kind of thing across the network if you have to.

no.

it requires acces to the serial console or baseband management controller or whatever terms have emerged.

have never rented a physical server w/o this.