News for hackers. Stuff that matters.
Re:Pages With Too Many Ads Above The Fold Now Penalized By G... by AdamTReineke on Friday January 20, 2012 @03:55PM (#3491184)
There was an article a while back about how Googlebot is likely a version of Chrome that fully renders a page to process it. [1] You're right, without a full browser engine, you could pull a whole host of tricks to make the page appear clean but then put in ads later. Since Googlebot is probably a full featured browser, it becomes a lot harder to trick.

[1] http://ipullrank.com/googlebot-is-chrome/

Re:Pages With Too Many Ads Above The Fold Now Penalized By G... by HNatWORK on Friday January 20, 2012 @05:14PM (#3491457)
There are definitely at least two different, shall we say classes of crawlers: The Googlebot and the Google Web Preview crawler. I don't know the extent of Googlebot's javascript parsing but the web preview crawler appears to parse javascript like plain webkit.

Here's a link where they show the UA for the Google Web Preview crawler: http://www.webmasterworld.com/search_engine_spiders/4353651....

I believe this crawler renders your pages for the preview snippets you get in search results when you hover over the arrow that appears on the right side of a result.

The previews I've seen would only look that way if javascript was being rendered and allowed to run for ~10-20s by my estimation--based on the progress of an animation that was previewed.

Re:Pages With Too Many Ads Above The Fold Now Penalized By G... by PaulHoule on Friday January 20, 2012 @04:18PM (#3491261)
I've seen a lot of evidence that Googlebot evaluates Javascript in a page. For one thing, I've got a site that has Facebook Connect on it, and I see Facebook's bot following on the heels of Googlebot. They work together just like those crabs that work in teams to cut up starfish...