News for hackers. Stuff that matters.
Re:Twitter has and internal root CA problem by ilyt on Monday March 13, 2023 @07:04AM (#35133998)
Create new root CA, ssh to machine, remove old certs, re-add machine to Puppet, sign the new CSR on Puppet master, then it will download new root.

Very simple operation... if you have working SSH access with root. If they don't, well...

Re:Twitter has and internal root CA problem by walrus01 on Monday March 13, 2023 @07:33AM (#35134239)
If you don't have ssh access with root, hopefully you have access to something like the underlying hypervisor, to do the equivalent of "sudo xl console vmname" on a xen dom0 to get the what is logically the same as a physical serial tty (or local vga+keyboard) console on the domU machine.

Or the VMware esxi emulated graphical console, etc.

Or if it's a bunch of bare metal machines, hopefully someone old-school in the organization thought to deploy 48/96-port rs232 console serial concentrators and wire them up to the db9 serial port on each physical server. And you didn't disable all local serial tty in your operating config.

Re:Twitter has and internal root CA problem by justsomeadvice0 on Monday March 13, 2023 @07:43AM (#35134309)
To my knowledge all modern DCs have out-of-band networks for this sort of thing that provide serial access to the BMC chip, nothing old school about that. Old school is having to submit a ticket to Jerry in the DC to walk the crash cart down to box 55AE, hook up a serial console, run diagnostics, and attach the output back to the ticket. You only have to deal with Jerry occasionally now, usually when the BMC or power rails fail.
Re:Twitter has and internal root CA problem by walrus01 on Monday March 13, 2023 @07:44AM (#35134318)
There's more than a few people who've decided the security risk of full console capable bmc is not acceptable - and if other fail over systems are engineered appropriately, not necessary at all. BMC/IPMI intentionally disabled/not connected to any network.

Anecdotally I have seen a number of low cost x86-64 pseudo blade setups similar to open compute platform design stuff which have no oob. If a unit fails it's pulled entirely and put in a work queue for someone to repair.

Re:Twitter has and internal root CA problem by ilyt on Monday March 13, 2023 @09:45AM (#35135642)
In both cases it's disruptive event as you have to reboot the machine to get into rescue mode (as you don't need the password)
Re:Twitter has and internal root CA problem by brazzy on Monday March 13, 2023 @07:44AM (#35134319)
> Or if it's a bunch of bare metal machines, hopefully someone old-school in the organization thought to deploy 48/96-port rs232 console serial concentrators and wire them up to the db9 serial port on each physical server. And you didn't disable all local serial tty in your operating config.

In a hacker folklore story this would 100% be the solution. And for some reason they'd have to use an original VT100 that some greybeard had lovingly restored at home.

Re:Twitter has and internal root CA problem by yuppie_scum on Monday March 13, 2023 @08:36AM (#35134863)
If they're in the cloud, it's pretty straightforward to re-mount the drive somewhere else and replace the SSH keys.
Re:Twitter has and internal root CA problem by ilyt on Monday March 13, 2023 @09:46AM (#35135655)
And if they have same template everywhere probably not even too hard to script.