thefarseeker

I'm Taryn's coworker I was involved in that part of the process very closely.

That statement is not entirely accurate. The SSDs certainly did die, but not _because_ of the 2019 upgrade, it just happened at the same time as the 2019 upgrades.

The major 2019 SSD issue was however that Intel and Dell kept passing the buck on who was to blame for not having 2019 drivers for the NVMe SSDs. Eventually I ended up having to bully Intel into giving us the drivers. We decided that "bullying Intel" was not an appropriate future tactic for getting the next version of the drivers, so that's another factor for staying on 2016 (which has drivers we've tested for years)

Not sure if trolling or... but fwiw at Stack Overflow's SRE team we use Go a _lot_. We just open-sourced our latest tool written in Go, DNSControl (https://github.com/StackExchange/dnscontrol/), and we have dozens of internal tools and software written in Go.

Not to mention other open-source tools like Grafana, InfluxDB, or Cockroachdb.

I agree with you that Java and JavaScript will reign supreme for years to come. Just pointing out that not all of us think that's a good thing, and there is plenty of variety out there, in the real world.

That was my takeaway too. If you ever had to change a spelling error in the front-end of a devopsy-style tool, then chances are you wrote some javascript, and bam: top of the list.

I'm in the camp of "I do all my serious work on a PC, but I take a MacBook to conferences because it's light and has good battery life".

So DNSMadeEasy made it all the way through the barrage of tests. I even wrote a library for their API so that we could integrate it into our DNS software (https://github.com/mhenderson-so/godnsmadeeasy), but at the end of the day their performance in certain regions was not good enough. In some countries they were measurably faster than R53, but in others they were measurably much slower.

EdgeCast were dropped due to pricing, and that there's talk of Verizon selling the EdgeCast services again.

DNSimple didn't make it to performance testing because they only had 5 POPs, as opposed to 20+ of other providers.

CloudFlare's DNS was consistently one of the fastest, you are correct about that. If you read my responses to other comments here, you'll find that we decided not use their DNS service because of some fairly pervasive API issues we had with it.

Author here. CloudFlare's DNS was measurably, and consistently, faster than almost every provider we tested. So I can concur with you in that regards. It was ultimately faster than Route 53 and Google Cloud (the path we ended up going).

However, when our relationship with CloudFlare ended and we moved to Fastly, one of the reasons we did so was unreliable lag with DNS updates at CloudFlare. Sometimes it would take minutes or hours for DNS changes to be affected in their system, and they could never provide a satisfactory reason why.

Author here. I agree. There are built-in mechanisms for doing this - AXFR and IXFR. However, these mechanisms were not really designed with this sort of scale in mind. You have to keep an up to date whitelist of all the servers that can talk to each other, and they would need to talk to each other on a non-anycasted address (otherwise the notify packet would go to just a single anycasted node).

Managing whitelists between multiple 3rd party DNS providers is likely to break frequently as servers move around, are added, removed, etc.

Interestingly, Hurricane Electric would have been one of our top choices if they had a first class API and a commercial SLA. Their ability to support zone transfers is admirable and did not go un-noticed. DNS Made Easy also supports zone transfers.

Author here. Google and AWS are anycasted services. We were not interested in running our own anycasted DNS due to the management overhead of doing so, and because the cost of anycasting our own services would be orders of magnitude more expensive than outsourcing that to an established DNS provider.

Author here. One thing I didn't cover in the post was about how to have Google and AWS honour their SLA's is that you have to use _all 4_ of the nameservers they provide. Because we're not doing that (only using half), we have to balance the chance of an outage versus the impact of an outage.

You are correct in saying that more empirical data could be used here. We might even end up changing our minds. I'm not much of a numbers person but I might pass this onto some of the people in our company who love solving problems like this.