Here's why: suppose some types are easier to answer than others. Maybe A and CNAME and AAAA are cached, MX is in the database but not cached, and NSEC3 requires computing a result. In order to correctly implement ANY (pre-RFC8482), you must convert it into all of those, and wait for the slowest one. An implementation like Firefox that asks for ANY because it just wants A and AAAA is wasting both the client and the server's time.

However, also, pre-RFC8482 that's not what ANY is defined to mean. A non-authoritative caching resolver is permitted to respond with only the records it has cached if something is in cache. So you're already getting a random subset of records. So the behavior is unhelpful for clients and also unhelpful for authoritative servers, and not in a way where the unhelpfulness trades off for helpfulness elsewhere.

That semantics is indeed useless.

And to begin with, applications should never be interested in all record pertaining to a domain; they want specific records. If I'm sending mail, I want a MX record for a domain. I don't want the SPF record unless I'm receiving mail which implicates that domain. If I'm making a HTTP connection, I don't want any of these; I want an IP address. So no semantics of ANY is easily justifiable.

On the other hand, we can't claim that this semantics is hard to implement.

It's hard to implement in a way that makes sense in the constraints of the world as it is (people asking for ANY when they have some meaning in mind other than the nonsensical specs) and especially in a way that doesn't enable DNS reflection attacks. In an ideal world, sure, you could enable ANY provided nobody either good or evil ever used it. And in fact Cloudflare says they did previously implement it just fine, by littering if statements everywhere.

The article gives some insight. For example, to quote:

    Should it forward the "ANY" query to authoritative?
    Should it respond with any record that is already in cache?
    Should it do some a mixture of the above behaviors?
    Should it cache the result of "ANY" query and re-use the data for other queries?

Pair that with inconsistent implemented behavior.

If we treat ANY as universally quantified, for absolute correctness/up-to-dateness we must always send the request upstream, because we are not the authority on all that may exist. However, we can sacrifice a bit of correctness with some reasonable heuristics. If we execute an ANY query for a given domain, we can cache the fact that we did such a query, in addition to caching the records that come back. Then, for some time after that, we may treat those cached records as being everything that is known about that domain; if another ANY query comes in for that domain, we give those records to the client. Furthermore, in an independent cache entry, we can associate the list of record types with that domain: it informs us that for that domain, records of those types are known, or have been known. When we get an ANY query for that domain, if we have this list, we can walk the list of types and query upstream for the missing records that have expired from our cache, rather than forwarding the ANY request. Then we serve the client a mixture of cached and fresh records.

Oh my. Multiple heterogenous but interacting caches? Heuristics? And if I understood it correctly, if any of the records you got from ANY are expired, you selectively query for new records from a list of cached types, not using ANY? What if the upstream added a record from another type in the meantime? Why have ANY at all then?

How is any of that "simple"? How do you transition the existing infrastructure to that behavior? Why?

> What if the upstream added a record from another type in the meantime?

Then we don't see it for a while until the type list is refreshed; that's the up-to-dateness case that is sacrificed.

> How do you transition the existing infrastructure to that behavior? Why?

If it's an alternative to ALL being rendered inoperative (and someone else is already taking responsibility for that extreme measure), I might have a lot of leeway that adds up to "try it and whatever the hell happens, happens".

No. Don't add complexity, state and attack surface with no reasonable cause.

I’m no DNS-expert but I can tell you missed AAAA, SRV and INFO, so clearly it’s not that easy.

"Say we have ..."

Yes, but now consider that you have to forward the query. If you unpacked the ANY to individual known types at the frontend, will you only forward A, CNAME and MX (which you yourself know about in your example), and omit AAAA, SRV and INFO? (And also be responsible for packing them back into a single reply, I guess?)

More correctly you should probably forward the ANY as ANY, so you have to do it slightly later than "up front". And then cache the individual answer records as if they came from individual queries.

I haven't fully thought this through, though, because: a) As you can see, it's not necessarily "simple"; b) it's not impossible to suss out something that makes sense, but the article at least claims that the resolver side is undefined so far, and several implementations do several different things by now. Giving up on ANY is probably for the best, at this point.

I just finished replying to your earlier comment.