Aarch64 on the other way...
The point was, that there will be not a problem with tagged pointers. The pointers have to be clean today when used, so those who use tagged pointers have a place in their code where they handle the cleaning.
Some very old distributed systems used 64 bit pointers where the upper bits told you what machine the data was on.
And we have Intel putting SSD card in DIMM slots. I expect there are semantics attached to those address ranges as well.
I dunno where we are after Spectre, but one of the microkernel architectures got its IPC speed by packing most processes into a couple of address spaces, only with different read write access. On preemption the TLB wasn’t invalidated (about half the cost), just modified.
These are the sorts of tricks you do when your address space is bigger than you will ever need. I expect to see more of this as time goes on. And if we ever see 128 bit pointers, expect crazy stuff like this to be de rigeur.
There are still things you could do, of course. For example, you could more or less NaN-box but rotate the bits so the tag is in the lowest bits instead of up in the exponent, then ensure all your pointers are sufficiently aligned. That would mean a perf hit on doubles to do the rotation to recover the double, but you might be right that this is not too bad in practice on modern hardware.