News for hackers. Stuff that matters.

A comment on Post-mortem for Partial Cloudflare Outage on October 25, 2022. Parent comment

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by systemvoltage on Wednesday October 26, 2022 @03:13AM (#33340701)
As an outsider without any knowledge of the company infra:

1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?

2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.

[ Parent | on HN ]
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by yjftsjthsd-h on Wednesday October 26, 2022 @03:22AM (#33340760)
In all fairness: If nothing else, a way to quickly deploy or revert is also a way to quickly break everything.
[ Parent | on HN ]