News for hackers. Stuff that matters.
Re:How Googlebot crawls JavaScript by anaolykarpov on Tuesday May 12, 2015 @05:34AM (#9530462)
I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()
Re:How Googlebot crawls JavaScript by netnichols on Tuesday May 12, 2015 @06:55AM (#9530766)
They probably don't care about that content.

My first guess would be that they snapshot the DOM in the JS tick immediately after window.onload completes. Maybe they have a short pause to let any fast timeouts or callbacks complete, but there's got to be a cutoff at some point (e.g. to stop an infinite wait for pages that continuously update a relative date). Of course, with their own JS engine, I bet they can get really fancy with the heuristics to determine when to take that snapshot.

Re:How Googlebot crawls JavaScript by KMag on Tuesday May 12, 2015 @08:32AM (#9531229)
Actually, we did care about this content. I'm not at liberty to explain the details, but we did execute setTimeouts up to some time limit.

If they're smart, they actually make the exact timeout a function of a HMAC of the loaded source, to make it very difficult to experiment around, find the exact limits, and fool the indexing system. Back in 2010, it was still a fixed time limit.

Source: executing JavaScript in Google's indexing pipeline was my job from 2006 to 2010.

Re:How Googlebot crawls JavaScript by blumkvist on Tuesday May 12, 2015 @11:37AM (#9532537)
What about AJAX? Does it load/read/index data after the fact?
Re:How Googlebot crawls JavaScript by andybak on Tuesday May 12, 2015 @06:37AM (#9530697)
A better example would be "how does Google index a page where an element changes sufficiently slowly that a user would see it post-change but a faster-than-real-time script would have a different experience"

Think of a long page where you need to read a few things as you scroll. You could game Google by timing it so that most humans would see content x but any script that ran at an unnatural speed would see y.

Re:How Googlebot crawls JavaScript by biokoda on Tuesday May 12, 2015 @06:12AM (#9530595)
Maybe they trigger timers immediately
Re:How Googlebot crawls JavaScript by kuschku on Tuesday May 12, 2015 @06:43AM (#9530723)
And how about server-side state?

Imagine you use setInterval to load a new paragraph from a server, and the server only provides a new parapgraph 1200ms after the first polling?

Re:How Googlebot crawls JavaScript by KMag on Tuesday May 12, 2015 @09:12AM (#9531442)
Each time we re-analyzed the page, we got more data and more URLs for the crawler to grab and have waiting for us the next time we analyzed that particular page. Of course, session-dependent content would get badly messed up, but that generally doesn't make useful (or at least repeatable) search results anyway.

Source: I primarily did JavaScript execution for Google's indexing pipeline 2006 to 2010.

Re:How Googlebot crawls JavaScript by fridek on Tuesday May 12, 2015 @06:52AM (#9530755)
Waiting for all one-time async operations to finish is not unfeasible. Async loops (like pinging server for possible updates) is a tad harder.
Re:How Googlebot crawls JavaScript by Rygu on Tuesday May 12, 2015 @06:43AM (#9530721)
Seriously, they don't have to. That kind of timeout makes the DOM element irrelevant.