I'm not sure. If that was the case, I don't think they'd put a hot ass team (including Lars Bak IIRC) on the Dart VM and spend millions on it.
I believe Google would very much like Dart (the VM) to succeed, but they know that realistically it can not for now, and so they invest in the JS translation in order for it to be a credible player.
As you mention, one way that Dark can make a dent, though, is on the server side. It's a decent dynamic language, that's an order of magnitude faster than Python/Ruby. With enough library support, it can get places that even node, tied as it is to the async callback model, might not. A decent web framework, for example, could have Dart see a lot of adoption.
That said, it's still too early to tell -- it's not even out of beta yet.
Furthermore, the compiler is responsible for resolving the dependencies your main program uses ("tree shaking" to remove stuff you don't actually use [1]), so there has to be a compiler even if your browser had a Dart VM built in.
[1] http://blog.sethladd.com/2013/01/minification-is-not-enough-...
I use Dart because it allows me to organize and test my code in a way that is very inconvenient with Javascript. Any speed improvements or IDE support is incidental, I have not yet had any speed problems in Javascript and I don't use their IDE :) The type-checking, library system, and the language itself is wonderful, though.
(Remember that you can write server-side software with Dart, too, so the VM can exist for that purpose without being some evil scheme to eliminate Javascript. Not that I would mourn the passing of Javascript :)