News for hackers. Stuff that matters.
Re:The case of the recursive resolvers: What happened during... by toomuchtodo on Monday November 29, 2021 @04:37PM (#29385610)
No tech company is infallible. All of them have outages, some lasting hours, even days.

Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless.

One day it might just be your turn to break production.

Re:The case of the recursive resolvers: What happened during... by tptacek on Monday November 29, 2021 @04:39PM (#29385625)
The subtext here isn't that Slack is bad at this (they are not), but that DNSSEC is somehow intrinsically unsafe (it probably is).
Re:The case of the recursive resolvers: What happened during... by toomuchtodo on Monday November 29, 2021 @04:49PM (#29385714)
I agree with your points about DNSSEC (disclaimer: I have not had the pleasure of having to implement it myself in infra), but was attempting to communicate that DNSSEC isn’t the only area of ops that folks get exposed to these sorts of unknowns or edge cases, and that no amount of resourcing enables you to avoid these issues. For Slack, it was DNSSEC. For Roblox, Consul. Facebook/Insta, software defined BGP. Akamai, DNS.

Perhaps I did not read the room appropriately. Mea culpa.

Re:The case of the recursive resolvers: What happened during... by Operyl on Monday November 29, 2021 @11:44PM (#29389055)
Did Roblox finally come out with their postmortem blaming Consul? As far as I know we just assumed it, but have had no update since October.
Re:The case of the recursive resolvers: What happened during... by toomuchtodo on Thursday December 02, 2021 @09:09PM (#29425807)