Circular dependencies can absolutely wreck you. For example, puppet could configure sudoers, and without puppet config being applied people who would normally expect access might not have it. So now you have to find a privileged ssh key for un-configured machines.
I would be surprised if twitter did not have a physical vault with a USB drive with a root SSH key on it. With that you can do just about everything.
I would be most terrified of machine churn. Auto-remediation systems or elastic capacity systems can result in lost capacity that can't come back until the configuration problem is resolved.
This is because the whole idea is that you have inaccessible, locked down Production servers that only Puppet (which is driven from a central, governed configuration management source) has authority to configure i.e. no SSH and no root access.
Thus leaving the only option being to physically visit each server at the datacenter and issue the commands.
[1] https://www.puppet.com/docs/puppet/5.5/ssl_regenerate_certif...
Which if we assume only a team of your size remains then it would take 300+ days.
That would mean no OS patches etc which would put them firmly in the crosshairs of the FTC.
[1] https://twitter.com/d_feldman/status/1562265193249390593
If it's not the case then Twitter is definitely in far more trouble because according to past engineers at least a few of their services needed manual intervention on a full scale reboot. And losing quorum in a distributed system is never pretty.
Anecdotal point of data: infosec.exchange hosts 30k users on 7 servers (https://infosec.exchange/@jerry/109374478717918484). That's a 1:5 ratio. Again, not the same usage and performance requirements, but I find it interesting.
Very simple operation... if you have working SSH access with root. If they don't, well...
Or the VMware esxi emulated graphical console, etc.
Or if it's a bunch of bare metal machines, hopefully someone old-school in the organization thought to deploy 48/96-port rs232 console serial concentrators and wire them up to the db9 serial port on each physical server. And you didn't disable all local serial tty in your operating config.
Anecdotally I have seen a number of low cost x86-64 pseudo blade setups similar to open compute platform design stuff which have no oob. If a unit fails it's pulled entirely and put in a work queue for someone to repair.
In a hacker folklore story this would 100% be the solution. And for some reason they'd have to use an original VT100 that some greybeard had lovingly restored at home.