I'd have a hard time using Emacs for a large project, especially one that is contributed to by many different people. IDE's just make life so much easier. The people who disagree with that statement are typically the Emacs (or VIM) gurus who have customized it so much that they would probably die if they had to use a stock installation.
Modes definitely add complication, but again, that's why Emacs is so customizable. It takes time, but you can set things up to where it's the best experience possible.
However this is trivially fixed because in Emacs everything is configurable. Keymappings are very easy to change (I'm using my own, completely custom ones, in my own keymapping minor-mode). There's also the "evil mode" reproducing the very superior (compared to Emacs) vim keybindings. So it's not true that they cannot be fixed. Not only can keybindings be fixed but a) they should and b) it's trivial to do so.
I'm using my own minor-mode which contains all my keymappings. Doing it that way you'll hardly ever have conflicting shortcuts.
evil-mode is an option too for people who want the (way better) vi(m) keybindings.
That being said, Emacs has already forced me to re-learn basic things like copy/paste and search/replace. I don't want to question my workflow on such a low level in 2014, when there is much higher-level IDE stuff to learn (code completion engines, refactoring, "jump to definition"). Alas, such features are not language-agnostic, they are tricky to set up and not standardized.
I find discussions about keybindings quite derailing from the higher-level discussions we should be having instead.
I agree with the language support. There might be better editors for every single language. But overall its the editor I like most of all I tested and used.
I don't know LISP very well and can't compare the speed to others. Yet emacs feels snappier than all the modern editors I've used. Probably just because it doesn't use fading effects and such nonsense.
And I'm not some beardy programmer from the punch card ages. Hell, the editor is older than me. But I do configure the shit out of tools I use every day. On that road, I haven't come to any limits in emacs.
I would never recommend it for a newcomer nor to people that want out of the box solutions for a specific domain.
The reason why I'm so upset is, if you are a newcomer and look at Emacs what you see at first glance on the outside is basically total crap. But then people tell you that the learning curve is high and everything is awesome if you just learn.
Well my experience was, the more I learned the more crap appeared. One layer of crap after the other. Luckily everything could somehow be fixed with a lot of configuration and scripts. But the points I mention above are the lowest layer which can not be fixed with just a bunch of lines of Lisp like in the previous layers. So even at the core Emacs is crap.