News for hackers. Stuff that matters.
Re:Problems with low DNS TTLs by JimWestergren on Monday March 29, 2021 @10:17AM (#26621936)
What is the use cases for having the TTL shorter than 5 minutes?
Re:Problems with low DNS TTLs by nathanaldensr on Monday March 29, 2021 @10:23AM (#26622055)
The article posits why:

Why are DNS records set with such low TTLs?

- Legacy load balancers left with default settings

- The urban legend that DNS-based load balancing depends on TTLs (it doesn’t - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can’t connect)

- Administrators wanting their changes to be applied immediately, because it may require less planning work.

- As a DNS or load balancer administator, your duty is to efficiently deploy the configuration people ask, not to make websites and services fast.

- Low TTLs give peace of mind.

- People initially use low TTLs for testing, and forget to crank them up later.

Re:Problems with low DNS TTLs by cortesoft on Monday March 29, 2021 @11:50AM (#26623635)
The DNS based load balancing isn’t a myth if you want to do any kind of load balancing that isn’t round robin. If you want to, say, send 10% of traffic to data center A and 90% to data center B, and you don’t want to use up 10 IPs to do that.
Re:Problems with low DNS TTLs by JimWestergren on Monday March 29, 2021 @10:25AM (#26622090)
But those are no valid use cases which was my question. So there are no valid use cases at all?
Re:Problems with low DNS TTLs by hiq on Monday March 29, 2021 @10:31AM (#26622169)
> Administrators wanting their changes to be applied immediately, because it may require less planning work.

Why is this not valid?

Re:Problems with low DNS TTLs by JimWestergren on Monday March 29, 2021 @10:40AM (#26622363)
But 5 minutes should be fine? It surprised me that so many has 1 second or 20 second TTL.
Re:Problems with low DNS TTLs by samvher on Monday March 29, 2021 @11:51AM (#26623646)
If I need to get a web service up and I can save 4 minutes by setting a low TTL when I configure my DNS record why wouldn't I?
Re:Problems with low DNS TTLs by gertrunde on Monday March 29, 2021 @01:17PM (#26625088)
Because you're pushing the cost on to someone else.

If your DNS hosting provider charged you per query (some do, especially when adding features like health checks & load balancing), then it might make a big difference.

Re:Problems with low DNS TTLs by fomine3 on Tuesday March 30, 2021 @01:11AM (#26631430)
Valid migration plan should be able to handle it without very short TTL.
Re:Problems with low DNS TTLs by gwittel on Monday March 29, 2021 @11:18AM (#26623023)
The major cases revolve around failure recovery, and traffic distribution. A 5 minute outage is not acceptable in many industries or at scale.

If a load balancer or DC fails we need to ensure traffic moves away fast. Similarly if you want to take a system out for maintenance or perform migrations.

Re:Problems with low DNS TTLs by marcosdumay on Monday March 29, 2021 @01:36PM (#26625372)
> A 5 minute outage is not acceptable in many industries or at scale.

Well, if that's the case, you better have your redundant systems on your normal DNS entries, because there is no chance you will distribute new entries over the internet in 5 minutes, whatever value you specify at the TTL.

Re:Problems with low DNS TTLs by toast0 on Monday March 29, 2021 @06:41PM (#26629212)
There are tons of things that don't follow TTLs, but a large majority of normal people traffic does.

Easily 90% of new connections will move following the TTL. Of course, some traffic got a DNS result once in 2003 and is going to use that forever. If it's important traffic, you can trace it and follow up with them. If not, you do the best you can and let the rest go.

Re:Problems with low DNS TTLs by JimWestergren on Monday March 29, 2021 @10:36AM (#26622289)
Found an earlier HN discussion about this question from November 2019: https://news.ycombinator.com/item?id=21437160
Re:Problems with low DNS TTLs by prophesi on Monday March 29, 2021 @10:32AM (#26622195)
A low DNS TTL for testing purposes is a valid use case.
Re:Problems with low DNS TTLs by majewsky on Monday March 29, 2021 @04:02PM (#26627445)
This is about production systems of large enterprises.

...

Please tell me you're not testing in production.

Re:Problems with low DNS TTLs by benmmurphy on Monday March 29, 2021 @10:41AM (#26622385)
AWS application load balancers have records with a TTL of 60s. Presumably they are doing it because they want the flexibility to change the IP addresses or the number of IP addresses dynamically. Seems like a reasonable use case.
Re:Problems with low DNS TTLs by krzyk on Monday March 29, 2021 @10:45AM (#26622450)
Maybe for those cases where one can get random IP from their ISP and has a e.g. hopto.org configured?

One would probably be OK if it was 5 or 10 mins, but it depends on what's behind that dns entry and how often ISP can change the IP.

Re:Problems with low DNS TTLs by abricot on Monday March 29, 2021 @12:20PM (#26624143)
Do any ISPs generally change dynamic IPs more often than modem/routers reboot?
Re:Problems with low DNS TTLs by majewsky on Monday March 29, 2021 @04:04PM (#26627472)
Some ISPs do it on a daily schedule. I know that the German Telekom rotates customer IPs every night at 01:45 AM, because a friend of mine is with them and that's the time when he drops from the video conference for a minute (if we stick around that long).
Re:Problems with low DNS TTLs by cortesoft on Monday March 29, 2021 @11:48AM (#26623602)
DNS based network load balancing. If you have two data centers, and you want to be able to dynamically and deterministically shift load between them, you want a short TTL so you can control the percentage of traffic going to each data center.
Re:Problems with low DNS TTLs by numbsafari on Monday March 29, 2021 @12:24PM (#26624207)
If you want to deterministically shift load, you use routing, not DNS, to manage your load.

That’s what is missing from this discussion.

Re:Problems with low DNS TTLs by toast0 on Monday March 29, 2021 @06:49PM (#26629283)
Anycast/BGP traffic engineering is not nearly as accessible as DNS based load balancing. There's several DNS providers you can use to add load balancing and health checks on top of your existing hosting, wherever that is (and you can do it yourself too). Using anycast for this means a specialty hoster, a third party in the data path between your clients and servers, or running your own ASN. I'm sure that gets more precise results than DNS, but it's also a lot more work, and it's harder to replace if the providers involved stop being a good fit.

Determinism isn't necessarily required either. Probabilistic shifting works fine mostly.

Re:Problems with low DNS TTLs by cortesoft on Monday March 29, 2021 @12:45PM (#26624556)
How would you use routing to balance load at that granularity?
Re:Problems with low DNS TTLs by numbsafari on Monday March 29, 2021 @02:46PM (#26626413)
Rather than get into a lot of details, here's some excellent starting points:

[1] Google Cloud networking in depth: Cloud Load Balancing desconstructed - https://cloud.google.com/blog/products/networking/google-clo...

[2] What is AWS Global Accelerator: https://docs.aws.amazon.com/global-accelerator/latest/dg/wha...

[3] Tumblr: Hashing Your Way To Handling 23,000 Blog Requests Per Second: http://highscalability.com/blog/2014/8/4/tumblr-hashing-your...

[4] Load Balancing without Load Balancers: https://blog.cloudflare.com/cloudflares-architecture-elimina...

Re:Problems with low DNS TTLs by cortesoft on Monday March 29, 2021 @04:54PM (#26628035)
Yes, I understand how anycast works (I work for an anycast based CDN).

The issue is that you can't do percentage based routing with anycast... in fact, you can ONLY do shortest hop routing with anycast (at least for WAN anycast). That means that, while different edge networks can go to a different datacenter, every individual edge network will hit only a single datacenter.

The key issue is that anycast is a very blunt tool. You are relying on your BGP announcements to route traffic, but you aren't actually in control of where a particular request goes.

Re:Problems with low DNS TTLs by EricE on Monday March 29, 2021 @01:40PM (#26625437)
Rather easily. There are routing protocols designed for such things. Far more reliable than trying to hijack DNS for load balancing.

Indeed the root DNS servers are not a single server but pools of geographically distributed servers via anycast.

Re:Problems with low DNS TTLs by cortesoft on Monday March 29, 2021 @02:09PM (#26625878)
Anycast doesn't support percentage based load balancing unless you control all the hops between client and server, which is almost never the case if you are serving the public.

Every request that comes from the same network is going to be routed the same way. Anycast works great for regional load balancing in general, but it doesn't work for subdividing individual networks.

Re:Problems with low DNS TTLs by codyb on Monday March 29, 2021 @10:23AM (#26622066)
I always was under the impression people lowered the TTL when updating records... which doesn't even really make sense since the change won't propagate until the previous TTL is overrun anyways.

Then you were supposed to update it to a longer TTL when your change had propagated.

So, I guess, after understanding things better... there is no use case really since if it's a new record your change will always propagate, and if it's an old record, lowering the TTL on update doesn't really matter since the old TTL will still be in effect.

Re:Problems with low DNS TTLs by aequitas on Monday March 29, 2021 @10:32AM (#26622199)
It could help to lower your TTL way before you plan to update. If your current TTL was 24h, you just update it to 5m 24 hours before you plan the actual change of the record itself. The new record can them be set to 24h directly (unless you want a quick turnaround for rollback).

It's still no guarantee all changes will propagate within 5 minutes. But it gives some ease of mind to know the bulk of change won't take a day.

Also a lot of people forget the negative caching of NXDOMAIN records which is set by the TTL of the SOA record. Which means that it will take a while for your new record to be resolved if you started querying before you set the record.

Re:Problems with low DNS TTLs by EricE on Monday March 29, 2021 @01:44PM (#26625496)
Yes - it’s called planning. Lowering TTL well in advanced is not a new concept and works very well if you have enough advanced notice from the application owners ahead of time :p
Re:Problems with low DNS TTLs by toast0 on Monday March 29, 2021 @06:51PM (#26629310)
Unfortunately, data centers rarely give 24 hour notice about catastrophic fires or provide advance notice about utility outages when the automatic transfer switch is going to fail.
Re:Problems with low DNS TTLs by mschuster91 on Monday March 29, 2021 @10:33AM (#26622215)
> It's still no guarantee all changes will propagate within 5 minutes. But it gives some ease of mind to know the bulk of change won't take a day.

Especially it gives you peace of mind that should stuff go badly wrong you can easily revert the change.

Re:Problems with low DNS TTLs by ru552 on Monday March 29, 2021 @10:36AM (#26622278)
You got the use case right, but I think you're still thinking about it wrong. In-use public DNS changes aren't something that just crop up and need to be done immediately.

Eg. Five days from now, I'm going to make an infrastructure change that will affect public DNS. Today, I lower my TTL on affected records. Five days later I make the public changes, still using the lower TTL. Once I am satisfied my change will stay in production, I modify TTLs to be more sane.

Re:Problems with low DNS TTLs by elp on Monday March 29, 2021 @10:59AM (#26622707)
In practice though other things come up and the admin doesn't get round to increasing the TTL back to sanity later because nothing is broken if they don't.
Re:Problems with low DNS TTLs by gnu8 on Monday March 29, 2021 @02:02PM (#26625762)
Sure but in a department where people are sysadminning reactively, they’re going to have that problem crop up everywhere. This type of thing wouldn’t be neglected in a well thought-out change process.
Re:Problems with low DNS TTLs by codyb on Monday March 29, 2021 @10:38AM (#26622329)
That makes a lot of sense, I've updated a lot of DNS records, but not generally in a planned manner.
Re:Problems with low DNS TTLs by dawnerd on Monday March 29, 2021 @10:32AM (#26622200)
I lower the try a few days before I’m going to make a big change. This is also how we did it in the past at a large corp I worked at.

Does it actually make a difference? I dunno but it just feels right.

Re:Problems with low DNS TTLs by irishsultan on Monday March 29, 2021 @10:31AM (#26622172)
You could lower TTL if you know a change is coming.
Re:Problems with low DNS TTLs by ilikepi on Monday March 29, 2021 @10:35AM (#26622264)
Exactly. The TTL is lowered in advance of the maintenance window, sufficiently far out to allow any entries with the old TTL to expire from most resolvers. Once the maintenance has been completed and validated, and sufficient time has elapsed to decide there are no issues requiring roll back, the TTL is raised back up to its stable value.
Re:Problems with low DNS TTLs by rebelde on Monday March 29, 2021 @11:03AM (#26622769)
Windows Update. To reboot a server, you need to take it out of production. With a TTL of 5 minutes, it can take an hour for (nearly) all users to stop using that server.
Re:Problems with low DNS TTLs by yjftsjthsd-h on Monday March 29, 2021 @11:27AM (#26623174)
Sorry - why would a 5m TTL take an hour to stop using? Shouldn't it be 5 minutes?
Re:Problems with low DNS TTLs by Twirrim on Monday March 29, 2021 @11:54AM (#26623702)
If things behaved nicely, yes. There's all sorts of weird DNS caching behaviour out there. It's not unusual to find folks with DNS servers / clients that are caching records for 1 hour+, and then of course there's people running super old versions of Java that used to cache DNS forever by default (before JDK 6). There's a very clear set of user that seem to cache for 10-15 minutes, regardless of any DNS TTL.
Re:Problems with low DNS TTLs by marcosdumay on Monday March 29, 2021 @01:38PM (#26625405)
You can't fix systems that ignore your TTL by specifying lower TTL values.
Re:Problems with low DNS TTLs by Twirrim on Monday March 29, 2021 @02:10PM (#26625894)
Sure. My general approach is to use lower TTL values (~ 5 minutes) and just accept that if people do dumb things, they just have to put up with things randomly breaking unexpected.
Re:Problems with low DNS TTLs by EricE on Monday March 29, 2021 @01:43PM (#26625467)
Good grief- you do not need to reboot the server; just flush the cache https://www.dnsstuff.com/clear-flush-dns-server-cache-window...
Re:Problems with low DNS TTLs by majewsky on Monday March 29, 2021 @04:05PM (#26627488)
They mean they're rebooting the server having the IP that's entered in DNS, not rebooting the client consuming that service.
Re:Problems with low DNS TTLs by linsomniac on Monday March 29, 2021 @11:10AM (#26622893)
We have a service that uses AWS Route53 health checks, and set the records to 60s TTL because if there is a problem at the primary service fails healthcheck, we want it to get the updated DNS records, which point to another data center, fairly quickly.

In our case, primary is AWS with a protection service in front of it, and secondary is our own servers at a data center. So something like VRRP wouldn't work.