News for hackers. Stuff that matters.
Re:Google’s decreasingly useful, spam-filled web search by dools on Thursday January 06, 2011 @02:27AM (#2074459)
Hey moultano, I was just thinking about this problem: how can I tell that a site is spammy? Overwhelmingly, they look really really similar. For example, if you search for "diy solar homes" you'll see a wonderful example of some really spammy sites - they popup book offers on load, they have this kind of template where they have big garish fonts and a whole lot of information laid out carelessly on the page.

Then there's the "what you need, when you need it" category, and then there's the "put your google search in the title even though the page has no relevant results" category (mostly software download sites - ie. try searching for "application to use nokia e72 with itunes" you get this site filebuzz in the top two spots that has a whole lot of ads and a bunch of crappy non-related downloads).

So if you add a "uniqueness" index - ie. find ways to "semantically tag" not just the textual content, but the layout and font choices etc. of particular sites, that will catch the blatant affiliate spam bullshit (diy solar homes, what you need when you need it etc.) and then just figure out a way to prevent those "file buzz" type sites from sticking my search term in the title tags (I actually have no idea how this is done) you'll eliminate like 95% of the spam.

Re:Google’s decreasingly useful, spam-filled web search by moultano on Thursday January 06, 2011 @02:45AM (#2074502)
The top result for [diy solar homes] looks pretty good. http://www.builditsolar.com/ Looks like it has a lot of resources, though I don't know where it gets them or whether it has any claim to them. Was this one of the bad results for you?
Re:Google’s decreasingly useful, spam-filled web search by dools on Thursday January 06, 2011 @04:34AM (#2074725)
BuildIt solar looks like a relatively genuine attempt at building an online resource for information about solar homes, as well as some kits they sell for themselves.

Likewise http://www.treehugger.com/ looks like it has genuine content - although it's obviously a little bit thin on the ground.

These guys are clearly a legitimate business selling a product (well, legitimate website anyway):

http://www.supremeheating.com.au/pool-heating-top/solar-pool...

Now compare those three sites, to these four:

http://www.diy-solar-power-for-homes.com/

http://www.energy4living.hottipsonly.com/solar-power-for-hom...

http://www.diysolarpower4home.com/

http://www.solarwindpowerguide.com/diy-solar-heating/

this one, too, links back to earth4energy - strikingly similar to "earth4living" above:

http://greenerhomediy.com/create-solar-electricity-build-diy...

then we have this incredibly reputable and highly respected forum whirlpool:

http://forums.whirlpool.net.au/archive/1539413

As a human I find it relatively easy to pick out the massively spammy sites amongst those results that are on the first page for the search "diy solar homes" - and I think that any time you were to get several sites that are similar in some set of ways to each other that rank for the same search term then they should be "de-ranked".

So for example, thousands of people use the same wordpress themes, you can't just say "they're all spammers". But if the top 5 results for a particular search all share some measurable characteristics, you could safely say "hmm there's something spammy going on here".

Re:Google’s decreasingly useful, spam-filled web search by catch23 on Thursday January 06, 2011 @04:04AM (#2074676)
http://www.diysolarhomes.com/ (3rd result & 4th result) definitely looks spammy. It's like the site owner just bought a bunch of keyword-domain names in order to get high rankings on search. Almost all their links look like affiliate links.

If you click anywhere on the page, you'll get a popup asking you to buy some book of theirs.

Re:Google’s decreasingly useful, spam-filled web search by GaryGary on Friday January 07, 2011 @10:14AM (#2079729)
Hi -- I run BuildItSolar. Its a non-commercial site for people who want to build renewable energy projects. Its a retirement hobby, not a business. Some of the projects are my own, but many are projects that people have built and sent in the details. Its a site that is of, for, and by DIYers :) Gary
Re:Google’s decreasingly useful, spam-filled web search by schwabacher on Thursday January 06, 2011 @03:13AM (#2074577)
I think they could find your search terms from the referring url, but I am not sure how they are able to get their pages with your terms into the search results and get the terms into their meta description.

They must just have compiled huge lists of relatively specific search terms and have pages against each? But I would think this would be easy to identify and downrank..

It is a puzzle :/