There are other issues as well. The biggest offender being the inflexible heap size, e.g. when you launch an asm.js module, you pass in an array buffer as the heap which cannot be resized. If you need more memory than this, you are SOL.
This wouldn't be that bad, except in practice it needs to be ridiculously small if you want to run on most machines. 256MB, which not much memory for a game at all, will fail to allocate, or kill the tab on about 33% of machines (being generous. For us its probably more like 50% of machines, but we make educational games that need to run on university machines, which are uh, on the lower end).
Either way, as a result, a great deal of the optimization time I mentioned was spent getting all of our games to run with under 128MB heap.
That said it's definitely possible if you're careful (I'm not sure this is quite as true for a voxel game though, but I could be wrong). My complaint wasn't about the idea of using asm.js, it was the idea that Unity would produce a good result. It just doesn't.
However, I remember that the first Unity HTML5/WebGL demos a few years ago were much smaller, a complete zombie FPS had a 3.5MB upfront download which is actually pretty decent.
There is no inherent size advantage of doing things in Javascript vs. cross-compiled to asm.js. For instance, three.js is 100kByte minified and compressed, that's in the same ballpark as the (admittedly simple) demos here: http://floooh.github.io/oryol/
Once you are over the static overhead for the C runtime and web-API-wrappers of roughly 30..50kByte, emscripten demos don't grow much faster than minified and compressed Javascript code.
[edit: typos]