Generally there's three classes of trolls (this is just my opinion):
Griefers are what you describe. People who are more interested in gaming the implied social norms of the community in order to destroy it. These people suck, but totally blocking them eliminates the second class:
Dissenters are people who just disagree with you, and considering nearly every open source project is started by opinionated dissenters it's just bound to happen. I believe that without these people though your project dies because your ideas stagnate and you never really deal with actual criticism until it's too late. Also, they may just be the vocal part of your user base that wants something you don't.
Finally, moderators who use their moderation powers to "indirectly troll". By this I mean people who unilaterally deny people's communications because it would interfere with their hidden agendas. If you go look at nearly every failed community, at the heart is someone like this.
What I think, and what we'll find out, is that what really drives people away is not the presence of all of these, but the dominance of one of these types of "trolls". My idea is to just balance them out and give the community a real vote on what is what.
I think usenet can speak to the problem of trolls ruinging the community. Which is why web forms came to dominate, despite usenet's superiority in technical features and access (most people had email clients before web browsers).
Usenet was ruined by a small number of people who were trying to ruin others reputations by posting all sorts of insane diatribes to the group. Plus the spammers. Uggg... the spamemrs.
Your assumption is that it's impossible to differentiate category one from category two. I see no reason for that to be true.
Here's a question for you: If you moderate a list, do you let subscribers see what you've moderated and why? If it's for the benefit of the mailing list community, then why not?
If I did moderate a list (which I don't), and there was an ability for subscribers to see the raw feed, I'd allow that. I am curious if your approach will work. It's similar to how things operate here, but I wonder if it can work with no moderators. (HN does have editors who monitor flagged articles and comments.)
When you don't have to worry about if someone's being a troll or not real conversation happens.
This allows the creation of lists whose entire purpose is trolling each other (i.e. trolls trolling trolls trolling trolls...). Or in general if I think that the 'troll' flag is 'censorship,' then I have the option to disable people (on my list at least) from using it.
edit: fixed a typo
Most of the places on the Internet where I hang out, the trolls like the community they are a part of and are rather uninterested in destroying it. They do their thing primarily to get an emotional reaction out of people, and the easiest people to get an emotional reaction out of are the people who are constantly emotional. Because the overlap between permanently unhappy people and the "good" people (the latter group consisting of, e.g., core devs, or mods, or various other "central" members of the community) is (for most types of communities that I'm interested in) pretty small, that's not really a problem.
On the other hand, turning 4chan loose on a Neopets mailing list would probably end badly for the community, yes.
Over at kuro5hin.org we experienced a 'trollocaust' as it were, wherein the trolls successfully drove all the emotional beautiful souls off to another site (hulver.com). If the critical mass of trolls grows too large, community can't really be sustained.
Trolls are the predators of internet communities. They prey on the sick, the weak, and the perennially outraged (mostly this last group) and, if they are successful, drive them away from the community. A community without perennially outraged people is better than a community with them.
The question is whether this benefit is worth the potential cost, which is probably something that almost entirely dependent on the nature of the community and the personalities of the trolls involved.
Edit: By "perennially outraged", I am referring to people who are constantly angry or miserable for no reason other than that such an emotional state is "what they do". They aren't trolls themselves, any more than a particularly unpleasant customer is a troll. They're just constantly unhappy and vocal about it.