Re:What happens when you update your DNS
by henrikschroder on Monday June 22, 2020 @07:09PM
(#23608017)
> So you check DNS - your cache infrastructure calls lime, which has updated and gives the correct answer, it seems like everything is fine
Been there, done that, got burned. If you're mucking around with DNS records that are going to be verified by someone else, never trust your local lookups, always try to verify through a fresh third party that everything resolves correctly before you submit.
Because oops, something went wrong, the verification hit a wildcard entry with a cache time of days, and now you have to wait that long before trying again, because that entry isn't budging from other resolver's caches...
Yeah, flushing the cache in your own recursive resolver doesn't flush them all over the internet.
The most egregious case I've seen was an Amiga site. The site went down and for several days reported that users would need to wait for the updated records to propagate and lots of loyal fans were insisting anybody who couldn't read the site was just being too impatient.
What was actually wrong? They wrote their new IP address as a DNS name in their DNS configuration rather than as an IP address. Once they fixed that it began working and they acted as though that was just because now it had successfully propagated.
On the other hand propagation is a thing when it comes to distributing modified DNS records to multiple notionally authoritative DNS servers.
This can be a problem for using Let's Encrypt dns-01 challenges for example, especially with a third party DNS provider.
Suppose you write a TXT record to pass dns-01 and get a wildcard certificates for your domain example.com. You submit it to your provider's weird custom API and it says OK. Unfortunately when you do this all it really did was write the updated TXT record to a text file on an SFTP server. Each of the provider's say three authoritative DNS servers (mango, lime, kiwi) check this site every five minutes, download any updated files and begin serving the new answers.
Still they said OK, so you call Let's Encrypt and say you're ready to pass the challenge. Let's Encrypt calls authoritative server kiwi, which has never seen this TXT record and you fail the challenge.
So you check DNS - your cache infrastructure calls lime, which has updated and gives the correct answer, it seems like everything is fine, so you report a bug with Let's Encrypt. But nothing was wrong on their side.
Now, unlike typical "DNS propagation" myths the times for authoritative servers are usually minutes and can be only seconds for a sensible design (SFTP servers is not a sensible design) so you can just add a nice generous allowance of time and it'll usually work. But clearly the Right Thing™ is to have an API that actually confirms the authoritative servers are updated before returning OK.