Re:Let's blame the dev who pressed "Deploy"
by seiferteric on Sunday July 21, 2024 @09:50AM
(#41025084)
If you think about it, any software update can make something bad happen. Maybe we just shouldn't be running bare-metal installs for critical infra in the first place. Just run a hypervisor taking snapshots daily or before updates and if it bricks a system it restarts into the last good image. Critical infra should be much more resilient and defensive than it is.
[ Parent
| on HN ]
Easier said than done, since a hypervisor is yet another moving part that can fail.
People were running Crowdstrike on AWS (which in most (all?) cases involves a hypervisor), and recovery was still painful for them.
That only works if it's self contained. As soon as you need to reach out and interact with other systems, version mismatches can still cause issues even if your system hasn't changed. Integration points and input are the biggest points of failure or vulnerabilities. A system that doesn't take input or integrate with others is super easy to maintain (of course most are practically useless too).
Re:Let's blame the dev who pressed "Deploy"
by hypeatei on Sunday July 21, 2024 @09:56AM
(#41025132)
That, or just simplifying these systems with clear boundaries between them. Why are flight status displays running Windows with endpoint security in the first place? Data could be loaded periodically from one authorized machine/job on the network with all other outbound/inbound network access restricted.
Re:Let's blame the dev who pressed "Deploy"
by alexlll862 on Sunday July 21, 2024 @11:29AM
(#41025803)
your hypervisor needs security updates too.