But for the tech ecosystem as a whole, there are huge downsides.
I feel this personally because a former employer got three patents from my work, then assigned them to that particular pool. Despite my personal belief that those patents should not be patents, they could potentially be asserted against some popular technologies and big companies, and if they did then I'd be legally obliged to testify. (This is not a mistake that I feel inclined to repeat, but a decade ago I was not as informed as I now am...)
Selling a patent to a troll obviously has huge downsides, as you said.
But, for instance, if you're part of the whole mobile patent lawsuit mess, then it is darned hard to tell the difference between a defensive lawsuit and an offensive one.
The only problem is the fear of creating such a monster while trying to do good.