I actually happen to use an IDE on a Java project every day at work. I think Java can be a difficult language to work with otherwise. It's verbose so you end up with an enormous amount of code and there's a very slow feedback cycle. These are areas where an IDE can really help.
I rarely find myself wanting an IDE when I'm coding in Perl, Python, Javascript or pretty much everything else I use. Then again, maybe it's because I haven't seen an IDE with much to offer other than code completion when working with these languages. I have almost zero need or want for code completion. For me, some of the best an IDE has to offer is the ability to find things quickly and refactoring. Many of them don't do a particularly good job at this with languages other than Java.
So what you mention can indeed be an issue, but hardly a reason to dismiss IDEs in general.