The "hardcoded memory limit" you see is Emacs stealing three bit of a 32 bit word as type tags for its dynamic type system.

It might be time for Emacs to switch to 64 bit words internally, should give plenty of bits left for buffer addresses after taking a few bits of for type tags.

The memory overhead when editing small files would be larger, but I haven't really heard people complain about Emacs' memory footprint lately.