a) we pushed a change that was not well thought out in its ramifications

b) we probably didn't lower the TTL on our zones several days in advance of doing this change

c) we reverted in a panic!

d) by the way, did you know that many ISPs' caching nameservers don't respect low TTLs and will hold onto zones longer than they should?

e) please go bother and waste the time of some first tier support rep at RCN or Comcast or CenturyLink, who certainly has the power to administer their DNS servers...

oh jeez.

In fairness, this part really is a dumb problem on the ISP's end.

The thermostat connects out every 30 seconds to send a temperature report, and to check for settings changes.

The smart TV, every 30 seconds, sends a compressed snapshot of the image currently being displayed, for ad generation.

The doorbell starts a new connection to stream video whenever movement is detected. The cameras do the same.

Each device was built without any thought to DNS caching. As such, every new connection is going to trigger a new DNS lookup. Multiply that by however many homes are in the area, and the load really starts going up. Especially because each lookup from the device causes a series of lookups to happen on the recursive DNS server.

However that's why when you do a migration you lower the TTLs and eat the cost of lookup requests. Wait until your previous TTL lease expires (which is 2x the TTL) then start the migration. After everything is known to be good you up the TTLs again. That way you have a way to "quickly" recover if issues arise.

Seems refreshingly honest to me. Slack screwed up, it's not anyone else's fault, but there's nothing Slack can do about it now, here's a workaround.