The problem is not the frequency with which it happens. The problem is that the way you get notified that it has happened is that IP packets suddenly start to get delivered to the wrong machine. If you're lucky, the net result will simply be a dropped TCP connection. If you're not lucky, pretty much arbitrarily bad things can happen.

Now, it is true that IP is not reliable, and so arbitrarily bad things can happen at any time and you do have to be prepared for those. The problem here is that there is no notification. Potentially bad things are happening here not because something has gone wrong, but by design. That, IMHO, is the very definition of Bad Design.

Normally, when you make a DNS change, you control the DNS and the affected end points. That way you know when the change is happening, and you can give yourself as much time as you need to make the transition in an orderly way. With ELB, the only guarantee you have is the TTL. With a 60 second TTL, that means you have at most 60 seconds to do the transition, and even that is only if you notice it when it happens (and the only way to guarantee that is to poll the name server constantly).

I like to think of the problem as an inherent issue with cache invalidation in distributed systems that has no straightforward solution in an Internet architecture. DNS problems greatly resemble what it's like to have dangling pointers, for another analogy.

I think the expected solution approach is a little like validating 2FA logins. To account for system variability, you accept that transactions can take up to a certain time and you allow multiple values (the similarities probably end there, admittedly). With a more advanced solution you would even advise clients to invalidate their own cache too (similar to load shedding by advising / redirecting clients - you can see this in 2FA logins sometimes where you're asked for the sequence after the once-valid key just entered). So I think you'd need to accept multiple prior CNAME resolutions to account for longer lived transactions and make sure each entry change will be valid only for so long. Being able to be notified of these changes programmatically would be really nice though specifically for AWS. Perhaps AWS Lambda or SNS could be leveraged for pushing AWS DNS state change notifications to your system?

One approach I've seen some folks do is to simply reload their nginx configurations across their backend nodes to refresh the cached entry. It's probably intractable without sacrificing a lot of theoretical availability with in a degenerate state of reloading upon each request when your CNAME changes several times a second. Some of my colleagues have experienced problems even with the nginx proxy_pass directive that most people say is the most recommended free solution and also recommended in the article.