Of course you can go lower, but there will be sacrifices. Brendan Eich actually addressed this same topic in response to some other thread I started a while back. I quote him below:

"Apart from syntax wins, you can't get much lower-level semantically and keep both safety and linear-time verifiability. Java bytecode with unrestricted goto and type confusion at join points makes for O(n^4) verification complexity. asm.js type checking is linear.

New and more concise syntax may come, but it's not a priority (gzip helps a lot), and doing it early makes two problem-kids to feed (JS as source language; new syntax for asm.js), which not only costs more but can make for divergence and can overconstrain either child. (This bit Java, pretty badly.)"

asm.JS carries you rather close to bare metal.