Search engine spam is actually a fairly solvable problem if you aren't in Google's questionable position of also selling the ads that make the spam economically viable.

They can do everything except the one thing that would actually hurt the search engine spammers right in the coin purse: Penalize websites for having ads.

> The goals of the advertising business model do not always correspond to providing quality search to users.

- Sergey Brin and Lawrence Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine

Yeah, that one aged well :P

I don't know about that. If I recall correctly, there was a time when Google was trying to tamp down on pages with excessive advertising, so SEO spammers just switched to making pages that superficially looked like normal informational pages, but the content was all either ad copy or spun text, and all their links went to products they wanted to sell.

I get where you're coming from, but my experience is that even many people making sites for fun stick ads on the site in the hopes of getting a bit of extra cash. Lots of fansites did so in the olden days, and many useful sites, wikis and blogs do so now, especially if they either use hosting that adds them or they get enough traffic they feel they need to.

I'm not saying block ads, I'm saying prioritize sites that doesn't have ads above those that do. Prioritize those that have some over those that have a lot.

> Penalize websites for having ads.

Oh ... this is such a good idea. I'm like tempted to try it and see what happens.

Is Wikipedia's prompt to donate (which they do quite often) an ad?

The whole premise of Google at the beginning was that the web is a collaboration, and we can measure that.

Then it was we can measure that and make money.

Now it’s just we can make money.

Whether that part is sinister or not, we know that we have a good number of bad actors, and from search engine results we can be sure that they have not developed a workable Byzantine fault tolerance mechanism to filter out the bad actors. Those who scream the loudest get put on a stage.

Companies that acquire other companies and talent will always change. The goal of making money over everything is the one thing all Class-C corporations have in common.

An AI could make the list, and pretty soon the gamers of the system would opt for really terse recipe sites.

It cannot be gamed. You wrote your own search engine, you wrote your own filters, you know how it works and noone else.

Whitelists that I wrote by hand also don't introduce new unexpected entries by the way :)

This instead could be more like RSS where as your crawler gets new sites, you get updates on new things, and you could filter in your crawler or in RSS client directly, doesn't matter.