The ideal solution would be to make it so that the easiest path for someone trying to abuse Google is simply to be constructive. If your abuse is what people want anyway, then you've won. It's sort of like capitalism: assume that people are nasty sonuvabitches who will lie and cheat their way to fame, money, and power, and then make it so the easiest way to get those is simply to make things that other people want.
Obviously, we're not there yet, and a lot of these spam pages are fairly useless to visitors.
If you take advertising out of the equation then the motives for producing general-topic content revert to what they were a few years ago: personal expression, free expert opinion, community discussion, journalism in search of subscribers.
Note that this doesn't require an all-new crawl/engine: just for an existing engine to offer an advanced operator that filters ad-drenched pages from results. Even just an operator that eliminated AdSense sites would be a big win for some queries.
I would suggest that the content farms' success in gaming specifically Google's algorithm was an inevitability (whatever the current state of the arms race) and the only thing that will weaken the effectiveness of their techniques is to expose their business model to a greater range of algorithms. If you have three or four search engines all working on slightly different principles, it becomes a lot harder to game them all with the same content, even if gaming any one of them would be trivial. In other words, competition in the SE space at the algorithmic level is something we sorely need to see.
In parallel, my suggestion for one new search engine to add to the mix: a crawler for unsubsidised content. That is, the results consist solely of pages that don't carry advertising of any kind. This wouldn't exclude ecommerce sites but would exclude most kinds of affiliate marketing. Subscriber-only sites could pay to be indexed at a flat rate, though guaranteeing that this fee wouldn't affect rankings might be tricky. Alternatively a journal-access style of subscription model could see the SE paying the content site owner when one of its paying users consumes their information.