"The compiler can always assume writes to memory via one object don't change values in memory accessed via any other object."
Then, if you know your code can overlap, you can use some invalidate function to mark that "memory accessed via this object was changed by some statement prior, make sure to re-read values cached in registers". Because that's the odd case where you're doing something unusual. In general, "The compiler is allowed to optimise things assuming that spooky action at a distance isn't happening". You already have something very similar for atomics, which might be hidden behind a standard library, but in any case are a sequence of "lock the bus, read/write, memory barrier", telling your CPU and your compiler that memory access cannot be reordered around the memory barrier.
Seems to me that that could only happen if you had some other bug clobbering the values of in and out, or you're passing something to the function that you shouldn't be. In either case, it's an error on the side of the programmer.
> If you plan for this from the start with your language, you can drive the costs much lower.
It shouldn't be the job of the language or compiler to make up for the shortcomings of the programmer besides maybe code generation optimization.
In fact, you can't express this in C (or, restricted, whatever), so I do wonder if your assumption that because it's UB it'll compile to the optimal code is true.
I'm talking about something else, the issue of 'in' and 'out' pointing into the stack, so writing to them changing the value of the local variables. That is basically treated as UB by every C compiler, and no compiler I am aware of provides any switches which would make it easier to handle, it's just horrible and your code behaves in different ways depending on the compiler and optimisation level.
Now let's imagine we want to make it defined. Then the statement '* out = * in' would have to become something like:
1) Write the current values of i, in and out to memory (in case * in overlaps with any of their locations.
2) Read * in into a register
3) Write *out from that register
4) Re-read i, in and out from memory, in case they just changed.
Now, you could make that more efficient by adding to the start of the function a check which looked to see if either 'in' or 'out' pointed anywhere near the stack, but there is still going to be a significant cost. If you plan for this from the start with your language, you can drive the costs much lower.
The main thing that separates C from assembler is that you do not have to worry about what values are in registers, and which are in memory. However, that means any code which writes to memory provides a place where this can make a difference.