It takes "so long" because of the disparate cache and TTL idea. That is, service providers believe the best thing to do is run large DNS caches for end users (and those providers can each set TTL's however they wish; they might not all be in agreement on what is an appropriate TTL). Of course, that's not the only way to run a DNS. But groupthink (or whatever you choose to call it) prevents any change in the status quo.

For example, if I said end users could run their own caches, use their own custom zones, or custom /etc/hosts files for sites they frequent, I'll bet the idea would quickly be shot down by the defenders of the status quo.

Yet it solves the "so long" problem. They can honour the suggested TTL's the authoritative DNS servers provide, or set their cache's min/max TTL's as they wish. Long or short. End user decides. Not all sites, including CDN's, keep changing their IP every few minutes. Once the user knows the closest IP for getting what she wants, no need for the DNS. Things become very fast when you don't have to constantly look up the same names, day after day.

But the real question is why are you asking why it takes "so long"? What is the real problem you are trying to solve? For example, are you needing to keep switching IP's, add CNAMES, or create more indirection? If so, why is that?