x86_64 disallows this because the 286 only used the lower 24 bits of a memory address, which made developer (ab)use the extra bits like you said. This causes all kinds of havok with some 486 variants and the Pentium, which did use the full 32 bits.

AMD's original x86_64 CPU (Opteron) only used the lower 48 bits of the address (I think modern cpus still do), so to prevent any future problems they disallowed using the upper 16 bits.