What do you mean here by memory layout? For instance, the order of fields in a rust struct can (theoretically) change by recompiling. It's not defined by the order of fields in the definition.
On a language level, high level APIs will necessarily contain details to things like (mut) reference, Box whatever. Which is not a problem at all, given the problem domain, but in my opinion it is not possible to make a both low and high level language at the same time (and it is not really needed either)
The memory layout will simply leak into the program architecture and will have to be altered on refactors — something which is transparent with managed languages.