Re:Cloudflare outage on November 18, 2025 post mortem
by eastdakota on Tuesday November 18, 2025 @07:38PM
(#45974320)
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
[ Parent
| on HN ]
Re:Cloudflare outage on November 18, 2025 post mortem
by anurag on Tuesday November 18, 2025 @07:48PM
(#45974420)
Appreciate the extra transparency on the process.
Re:Cloudflare outage on November 18, 2025 post mortem
by philipgross on Tuesday November 18, 2025 @08:26PM
(#45974751)
You call this transparency, but fail to answer the most important questions: what was in the burrito? Was it good? Would you recommend?
Re:Cloudflare outage on November 18, 2025 post mortem
by eastdakota on Tuesday November 18, 2025 @08:43PM
(#45974882)
Chicken burrito from Coyo Taco in Lisbon. I am not proud of this. It’s worse than ordering from Chipotle. But there are no Chipotle’s in Lisbon… yet.
Re:Cloudflare outage on November 18, 2025 post mortem
by luckys on Wednesday November 19, 2025 @05:33AM
(#45977946)
There's a lot of good food places in Lisbon that you might not be familiar with yet. Enjoy your stay
Re:Cloudflare outage on November 18, 2025 post mortem
by dentemple on Wednesday November 19, 2025 @10:30AM
(#45980796)
I DON'T see this as transparency. There is ZERO mention of the burrito in the post-mortem document itself.
0/10, get it right the first time, folks. (/s)
Re:Cloudflare outage on November 18, 2025 post mortem
by jofzar on Tuesday November 18, 2025 @08:51PM
(#45974944)
> I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did
Damn corporate karma farming is ruthless, only a couple minute SLA before taking ownership of the karma. I guess I'm not built for this big business SLA.
Re:Cloudflare outage on November 18, 2025 post mortem
by dentemple on Wednesday November 19, 2025 @10:28AM
(#45980773)
We're in a Live Fast Die Young karma world. If you can't get a TikTok ready with 2 minutes of the post modem drop, you might as well quit and become a barista instead.
Re:Cloudflare outage on November 18, 2025 post mortem
by ynx0 on Tuesday November 18, 2025 @10:27PM
(#45975516)
How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups.
Thanks for the insight.
Re:Cloudflare outage on November 18, 2025 post mortem
by eastdakota on Wednesday November 19, 2025 @07:33AM
(#45978825)
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.
Re:Cloudflare outage on November 18, 2025 post mortem
by noname120 on Wednesday November 19, 2025 @12:58PM
(#45982600)
Could you elaborate a bit on how going to law school helped? Was it because it made it easier for you to communicate and align with your CLO?
Re:Cloudflare outage on November 18, 2025 post mortem
by tinfoil3843 on Thursday November 27, 2025 @10:09AM
(#46069933)
Probably because he could check legalities of a release himself without council. It is probably equivalent to educating yourself on your rights and laws so if you get pulled over by a cop who may try to do things that you can legally refuse, you can say no.
Re:Cloudflare outage on November 18, 2025 post mortem
by richardlblair on Tuesday November 18, 2025 @10:32PM
(#45975554)
A very human and authentic response. Love to see it.
Fantastic for recruiting, too.
Re:Cloudflare outage on November 18, 2025 post mortem
by anitil on Tuesday November 18, 2025 @11:37PM
(#45975906)
> He didn’t reply after a few minutes so I did
I'd consider applying based on this alone
Re:Cloudflare outage on November 18, 2025 post mortem
by realaaa on Wednesday November 19, 2025 @01:05AM
(#45976361)
that's very cool, thanks
Re:Cloudflare outage on November 18, 2025 post mortem
by jeffrallen on Wednesday November 19, 2025 @01:49AM
(#45976564)
A postmortem postmortem, I love it. Transparency to the power of 2.
Re:Cloudflare outage on November 18, 2025 post mortem
by tuetuopay on Wednesday November 19, 2025 @06:38AM
(#45978362)
> I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate.
I'm so jealous. I've written postmortems for major incidents at a previous job: a few hours to write, a week of bikeshedding by marketing and communication and tech writers and ... over any single detail in my writing. Sanitizing (hide a part), simplifying (our customers are too dumb to understand), etc, so that the final writing was "true" in the sense that it "was not false", but definitely not what I would call "true and accurate" as an engineer.
Re:Cloudflare outage on November 18, 2025 post mortem
by auto on Friday November 21, 2025 @12:01PM
(#46006276)
I'm not sure I've ever read something from someone so high up in a company that gave me such a strong feeling for "I'd like to work for these people". If job posts could be so informal and open ended, this post could serve as one in the form of a personality fit litmus test.
Re:Cloudflare outage on November 18, 2025 post mortem
by tinfoil3843 on Thursday November 27, 2025 @10:09AM
(#46069938)
I really appreciate this level of transparency. Thank you for being a good person in such a powerful position in the world.