The OS only let you alloc in large granules, like 4k or 16k, and the vast majority of the methods are significantly smaller than that, meaning a JIT must colocate multiple methods in the same allocation block or waste a significant amount of memory.
We could get around that by remapping memory between read/write to read/execute and have the OS solve the problem for us. Except for a couple of small details, modifying a memory mapping is very expensive and we're, well, in the performance business, and that mono is multi-threaded so one thread might be executing code from the exact page we just made non-executable.
This approach, IIRC, was tried by Firefox as it has some security advantages, but discarded due to the measurable performance impact - and they don't have the second problem as JS is single threaded.
Full Disclosure: I'm part of the Mono team.
Things get a lot more complicated when it comes to code patching, but the principle is similar.
Or is the flush guaranteed to flush all cores caches? That would be a fairly unusual design.
And the "Firefox (Ion)" and "Firefox (non writable jitcode)" lines on https://arewefastyet.com/ seem to coincide...
But yes, actually making this work in practice is not at all simple.
> modifying a memory mapping is very expensive
It used to be true that operations on memory mappings were appallingly expensive. However, the advent of virtualization has driven a significant change in performance. IIRC, ARMv8 has a TLB invalidation operation that is per-entry, addressed by the virtual address being invalidated. You don't need to flush the entire TLB cache.
In which case a manual flush of the icache is needed, hence this problem.
* allocate read/write buffer
* JIT instructions into it
* change mapping to read/execute
* run the JITted code
Then the kernel manages flushing the data caches on the mapping change, and Mono gets to wrap a Somebody Else's Problem field around it. It sounds like they are instead:
* allocate read/write/execute buffer
* JIT instructions into it
* manually flush relevant data caches (with an assumption that the cache line size is constant)
* run the JITted code