You are aware that programmers have been doing this in essence since Assembly Language was invented right? Programming directly in binary is just too much for our human minds to take.
That said, you are much more correct than you realize. Even programming a "simple" or "direct" translation, like compiling from C to object code is rife with pitfalls...
In general, as long as the compiler is bug free, you should be fine. Its just that never running into a compiler bug is much less likely when you're using very new code. Code is often buggy after all ;)
I see from another comment that you've used GWT. This may be why our experience differs. Java and Javascript are so far apart (ironic given their names) that to generate one from the other is inevitably going to be complex. We use Parenscript, and the semantics of Lisp and JS are much closer. As a result, I've been consistently surprised by how suitable JS has proven as a target language and how simple the whole approach is.
We've even extended Parenscript into a pretty expressive subset of Common Lisp compiling efficiently to Javascript. We never set out to do this - it just spontaneously evolved - and it's a small subset, nowhere near all of CL. But it covers enough that we can write large portions of our app to run both on the server and in the client, which turns out to be valuable sometimes.
That being said, we pay close attention to the generated Javascript, so I agree with your suspicion of X-to-JS compilers in that sense. What we do is a semi-abstraction.
But no matter what, you brush up against the fact that when the languages are just plain different, you have to be careful. And if you know Javascript is different from Language X, and can debug the difference, why are you writing Language X and not Javascript?
Honestly, most of what Lisp does that makes it unique (first class functions that aren't ugly, macros), are either availabe (first class functions), or something that the compiler can take care of for you, so it seems that Lisp is actually a scarily-good-fit for language to compile to JS. (Assuming no encountered bugs that is ;))
Actually, there is one big impedance mismatch: null. The semantics of Lisp and JS around null-zero-false are incompatible, and it leads to annoying bugs. I'm still looking for a good solution to that problem. The other big thing is that JS uses returns everywhere and Lisp doesn't. It might be possible to have the compiler add those in in the right places, but we haven't tried that yet.
Also, to answer your question:
And if you know Javascript is different from Language X, and can debug the difference, why are you writing Language X and not Javascript?
... the answer, when X is Lisp, is that the benefit of the additional abstractive power is much greater than the cost of the impedance mismatch. Both sides of that equation would vary greatly depending on X, which is just a way of repeating what we've already discussed :)
(Actually, I recommend sitting down and writing Javascript without a library, just so you have an idea of what works and what doesn't across browser. It was somewhat surprising for me to be sure.)