Re:How Meta Patches Linux at Hyperscale
by exitheone on Sunday December 03, 2023 @04:42AM
(#38505946)
That's still ridiculously slow.
I'd expect them to have hundreds of Microservices. Each one of those should be able to handle a random restart at any point in time so they should absolutely be able to restart 100s of servers concurrently without major disruptions.
Hell on Facebook scale a whole-Datacenter going down should not cause service disruptions.
This does assume that nothing is getting broken along the way.
Taking 45 days is probably more about caution and resolving issues systematically rather than pushing a big button and hoping you don’t cause issues.
I’d expect them to have thousands of microservices - and you only have to find a way to break one to cause big issues.
Re:How Meta Patches Linux at Hyperscale
by exitheone on Sunday December 03, 2023 @05:45AM
(#38506132)
Regular random crashes should be exercised regardless at Facebook scale. Not being resilient to that would be very unprofessional.
Re:How Meta Patches Linux at Hyperscale
by BlackFingolfin on Sunday December 03, 2023 @05:53PM
(#38511609)
That's not 15 minutes per host; that's 15 hosts per minute.
That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and marked as healthy)
"Oh but you can batch this" sure, but you don't want too much of a big batch that will make your service slow or want to risk shooting yourself in the foot - like rebooting your whole control plane then figuring out it doesn't work like that
(The 45 days is probably an estimate as well, I'm not sure they actually do that server by server)