Trouncing patents now would only give the already successful less to worry about, and would force the budding minds into the dirt.
I, for one, am so anti-capitalist that I don't use a single piece of proprietary software, and I own only what I can carry. I'm an idealist with the causes that I can control (personal use) - I'm a realist when I want to actually see change in the right direction (political voice).
I love open source software so much that I gladly stake my whole business model on it, to the point of wishing to experiment without the protection of copyright.
Those patent entrepreneurs wish to live in a world that would take my property right and freedom away in favor of their monopolistic business models.
This is liberty versus economic security.
I rather be a free dog who don't know what his next meal at, then an enslaved dog who get a small but a sure pittance.
And you're right, we must be somewhat disparate in our thought - If I were the architect of the world - I would have it that material property is the only sort that qualifies, and that nobody can make a living except by their work (or by the services they work to offer).
Assume that the absence of patents could send some would be lone inventors to poverty, while allowing the ideas they would have found to be found anyway. That's probably not OK for most people. However that's also a net benefit for society, and so is most probably OK. But "benefit for the society" is a vague and abstract notion. The "lone inventor" is something you can understand, feel, identify yourself with. I think a good argument against software patents would have to make people cry just as much.
The only patent holders large organisations fear are non-practicing entities (aka trolls) because anyone actually trying to build a product can be undermined in any one of a thousand ways.