That's the idea behind zero-copy serialization protocols like Cap'n Proto [1] and FlatBuffers [2]. The idea is that they define a particular mapping from memory layout to data model, and then provide bindings to that mapping from multiple languages. If you write them into, say, shared memory or a memory-mapped file, then multiple programs in multiple languages can access them, effectively for free.

There still remain a lot of real-world gotchas with this. There are compromises in versioning & upgradability; while both these protocols are designed so you can add new fields in a backwards-compatible way, they also bloat the memory requirements when you modify a schema enough. Code complexity is very high, which means the number of implementations is fairly limited. And many managed languages don't do well with direct memory addressing (Java, V8, and C# are particular offenders), so oftentimes the speed benefits are lost in translation.

But people have certainly been thinking about this issue.

[1] https://capnproto.org/

[2] https://github.com/google/flatbuffers