I guess there could be some fun workarounds with ptrdiff_t instead of pointers. "My child is 1000 bytes that way in memory" etc.

That's a pretty classical file serialization trick to pack a bunch of heterogeneous data in one file. A header/manifest starting the file, with an array of names, offsets, and sizes.

Actually works somewhat well for mmap'ed access, too, by using a macro or function to "dereference" "pointers" (offsets).

Or even an OffsetPtr class with overloaded arrow and dereference operator, if you swing that way. :)

What’s the runtime performance of that in-practice? Does x86, x64 and ARM have support for relative offset pointers without any performance impact? And do compilers use them?

Yes, address generation can at the very least do [reg + constant], and on x86, much much more. These are bread and butter optimizations constantly used by compilers all over the place, because this is how field access works.