Security is doable because BrowserVM does not need to do everything. It isn't generable purpose like the JVM, it is an embedded VM like Lua. Lua has great security, as far as I am aware. I expect that having a secure VM is no more difficult than having a secure ECMAScript implementation.
I think we want low-level execution primitives (i.e. assembly), which pretty much forces a VM, because it gives people flexibility to use whatever language they want. However, even the minimum of keeping JavaScript and having the page be a large canvas with (fast) drawing primitives would sufficient. Writing windowing systems and UI toolkits is a solved problem. I think the browser should, at minimum, provide primitives for drawing widgets, so that webapps can take advantage of OS-native widgets. A well-designed toolkit, though, would be handy. And here we have a number of examples. Qt is very well-designed and a pleasure to use. Cocoa is a little clunky, but effective. Even Java/Swing was fine (aside from horrid aesthetics). We know that Microsoft-style MFC is a bad idea, the Win32 callback idea is clunky, and Android is a mess. In particular, a constraints-style system seems to be clunky. It was clunky in Motif, clunky in Android, and even Apple's version doesn't seem all that great. Constraints seems to suffer from the same problem of HTML: describing the system is clunkier than just telling the system what to do. Hence, Qt and Swing's layouts are a lot easier to reason about.