Maybe its something about sharing memory with the js that would introduce serious vulnerabilities so they can't let wasm code have access to everything.

The only way to remove Js is to create a new browser that doesn't use it. Fragments the web, yes and probably nobody will use it