I think who they end up driving away is mostly determined by the kind of community and the kind of trolls.

Most of the places on the Internet where I hang out, the trolls like the community they are a part of and are rather uninterested in destroying it. They do their thing primarily to get an emotional reaction out of people, and the easiest people to get an emotional reaction out of are the people who are constantly emotional. Because the overlap between permanently unhappy people and the "good" people (the latter group consisting of, e.g., core devs, or mods, or various other "central" members of the community) is (for most types of communities that I'm interested in) pretty small, that's not really a problem.

On the other hand, turning 4chan loose on a Neopets mailing list would probably end badly for the community, yes.

4chan users are not trolls. They're griefers and sociopaths. Like you said, trolls are generally part of a community, or make an attempt to get a rise out of people through détournement, not through outright antagonism.

Over at kuro5hin.org we experienced a 'trollocaust' as it were, wherein the trolls successfully drove all the emotional beautiful souls off to another site (hulver.com). If the critical mass of trolls grows too large, community can't really be sustained.