A terrible, horrible, no-good, very bad day at Slack

Re:A terrible, horrible, no-good, very bad day at Slack by MapleWalnut on Tuesday July 07, 2020 @10:39AM (#23759728)

> It’s worth noting that HAProxy can integrate with Consul’s DNS interface, but this adds lag due to the DNS TTL, it limits the ability to use Consul tags, and managing very large DNS responses often seems to lead to hitting painful edge-cases and bugs.

I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight?

It seems like their service that broke would not be needed if they went the DNS route.

[ Parent | on HN ]

Re:A terrible, horrible, no-good, very bad day at Slack by blyry on Tuesday July 07, 2020 @11:01AM (#23759953)

Pre 2.0 there were a few bugs with SRV discovery, maybe they adopted early and got bit? Just an anecdote but we've been using it since 1.9 without issue. Massively different scales though.

Pre k8s and before srv support we used consul template in prod as well but it always scared me, seemed like too many moving pieces for what should've been a simple system.

[ Parent | on HN ]

Re:A terrible, horrible, no-good, very bad day at Slack by blyry on Tuesday July 07, 2020 @03:31PM (#23763086)

I asked internally and figured out the gotcha that bit us: default dns payload size is 512b, which is enough for a few backend hosts but for sure not 12 or 30. Limit is 8kb, which probably wouldn't work for whatever slack is doing.

https://cbonte.github.io/haproxy-dconv/2.1/configuration.htm....

Because DNS records come back in random order for each response, those truncated dns responses caused the backend slots to constantly rotate between different pod instances. Haproxy was graceful about the rotations, but it showed up as suddenly very strange latency / perf numbers when a backend was scaled up to say 10 instances from the normal 3

[ Parent | on HN ]

Re:A terrible, horrible, no-good, very bad day at Slack by user5994461 on Tuesday July 07, 2020 @11:10AM (#23760040)

Dynamic DNS support is only since haproxy 1.8, maybe 2 years old give or take. Slack infrastructure must be older than that so they improvised.

There were few bugs in the first implementations, should be good now. Slack don't mention their scale but I can imagine some UDP/DNS edge cases if there are hundreds of instances behind one domain.

[ Parent | on HN ]