Living in a civilized society means that we sometimes have to play by rules we don't like (until we can convince everybody else that it's time to change the rules).
My point is that–especially in the world of software–this general consensus does not exist for the patent industry, (that a large majority of everybody else is already convinced that it is time to change the rules,) and that this general dislike for the rules is the real source of the anger directed at patent trolls. This aligns with your earlier point about there being no difference between outsourced and in-house torture, in that I would say that people are just as upset about Microsoft's, Oracle's, and even Google's patent portfolios as they are about Intellectual ventures. The only difference between IV and those others is that they manage to win people over (or at least mollify) with the rest of their work.
Edit: I would say that seven_stones cutting straight to the morality of software patents in general pretty well illustrates my point here.
This is actually an interesting point. Both collection agents and patent trolls go about asking people to pony up the payment for goods (to put it colloquially) and they both do it on behalf of a third party, for whom doing it by themselves will involve a lot more of a hassles.
The distinction comes in the fact that in the case of collection agents, the payments owed to the goods are seemingly legitimate or at least socially accepted (perhaps collection agencies that come about collecting payments after those subscription-renewal or hidden-monthly-fee scams might be construed as as much a nuisance as patent trolls). I guess socially, asking payments for ideas only seems reasonable if the persons going about asking for the payments themselves come up with the idea.
It is a little weird and subtle. Can someone come up with a better distinction?