What you seem to be missing with re: asm.js is that, while the JIT to native code gets you your super-fast integer operations, it's still critically incomplete with regard to memory access. Every single individual memory access has to be bounds checked or pushed through some other indirection inside the runtime. Google demonstrated similar ideas with NaCl, which achieved safety with a similarly restricted native code and a just-in-time verification step. Even if these memory accesses could be made as efficient as those performed by the CPUs access protection, you're still not gaining anything you don't already have.

Regarding context switches: A full CPU context switch on x86 (not to ring1 but between two arbitrary points within a single userland address space) takes a few dozen instructions and about 40-80 cycles. A single cache-line miss resulting in a load from main memory on the other hand takes at least twice that (~200 cycles). Again, hits from jumping around in memory will dominate.

How significant is a 20% overhead from virtual memory? Probably about the same as getting 1% more of your memory accesses back in to high level caches.