Actually works somewhat well for mmap'ed access, too, by using a macro or function to "dereference" "pointers" (offsets).

Or even an OffsetPtr class with overloaded arrow and dereference operator, if you swing that way. :)

What’s the runtime performance of that in-practice? Does x86, x64 and ARM have support for relative offset pointers without any performance impact? And do compilers use them?

Yes, address generation can at the very least do [reg + constant], and on x86, much much more. These are bread and butter optimizations constantly used by compilers all over the place, because this is how field access works.