I was declaring data structures in shared mmap-ed memory as volatile so compiler does not optimise it away. How would the compiler know not to optimise away operation between memory fences? A fence to me say something about the guarantee of the order in which operations are seen. The order wasn't the problem, write never making it to the other side was the issue...
If the compiler is smart about its memory barrier intrinsics it will make sure to reread anything after the read barrier and write everything before the write barrier. This will speed things up by not having to use volatile. Between the barriers it can cache reads and inside registers and optimize out multiple writes like normal.
If the compiler can't do that then you need to use volatile. Volatile was meant for I/O where every read has to hit the bus. So if you're forced to use it then you just copy the volatile variable into a non volatile one for use inside your mutex to get some speed back.