News for hackers. Stuff that matters.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by hayst4ck on Wednesday October 26, 2022 @03:39AM (#33340877)
I don't know how cloudflare is configured, but doing anything fast to caches is generally a really bad idea. Caches often protect upstream data stores from overload. If an upstream data store becomes overloaded with too much traffic, then all requests that use that data store will start to slow down. As requests slow down, they will either timeout and fail resulting in major error spikes or they will start to consume all the resources of the service making requests.

So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.

I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.

Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.

There are no comments on this comment yet.