Re:Google says there are now 10^12 unique URLs. I am proud t...
by aristus on Monday July 28, 2008 @01:13PM
(#259272)
You detect them like Unix detects a symlink loop: it punts on the problem and just errors after 8 symlink traversals.
The equivalent is a crawl depth limit, which could be a hard limit (dumb) or a function of page-rank (smart) and the trustworthiness of the inbound link (smarter) and also the data quality & diversity of the traversed pages (best).
There are very good reasons why Googlebot seems to hit you from one IP at a time -- it's a long-running thread that is making all sorts of decisions about your site as it crawls.
Tackling the data quality & diversity of the traversed pages (best):
Producing English text with the 40 bits, by driving a generative grammar or a markov/travesty generator, would make it harder for Google to detect that the pages are auto-generated. It's unlikely to infer the function f(URL) -> text (or even to attempt it), but would limit the recursion for the other reasons you mention.
(guessing) sites like hackernews are indexed primarily by recursion (few direct inbound links to specific stories).
Re:Google says there are now 10^12 unique URLs. I am proud t...
by aristus on Monday July 28, 2008 @02:57PM
(#259443)
(guessing) sites like hackernews are indexed primarily by recursion (few direct inbound links to specific stories).
Correct. Notice that it is difficult to find old HN comments on Google, since after a while there are no short paths from the home page to them. In practice & all else being equal (quality, length, spamminess, speed, age, uniqueness, PR, etc), the maximum depth a page can afford to have is about 6 or 7.
Please log requests from the Google and Microsoft bots and let us know how long it takes the respective bots to figure out that every page is the same :-)
Re:Google says there are now 10^12 unique URLs. I am proud t...
by snprbob86 on Monday July 28, 2008 @03:34AM
(#258873)
So it looks like Google grabbed about 40 links before giving up? I wonder what a good "score" is? At first, I'd guess less is better, but too few might be running the risk of throwing out potentially good pages. Too many, and the bot is just wasting effort. The 40 score could vary as well based on parallel conditions assuming many bot instances are sharing a task pool. Be sure to post the Microsoft results if/when they crawl you.
Re:Google says there are now 10^12 unique URLs. I am proud t...
by brianr on Monday July 28, 2008 @07:37AM
(#258981)
Looks like it hit all of the links on the first page (there are 8x5 boxes = 40) and didn't find anything interesting, so it didn't crawl any deeper. If the second-level pages had more interesting/unique content, I bet it would've kept going.
Re:Google says there are now 10^12 unique URLs. I am proud t...
by attack on Monday July 28, 2008 @12:01AM
(#258707)
They actually did explicitly say that this measurement ignores URL's which are non-unique enough or of insignificant worth as best that they could determine. But ah well, I guess it's just for fun.
Re:Google says there are now 10^12 unique URLs. I am proud t...
by lpgauth on Sunday July 27, 2008 @11:00PM
(#258658)
I guess this goes to show how this metric is useless on it's own but if you compare with other previous years then you can understand how far the web as evolved.
http://en.wikipedia.org/wiki/Spider_trap
.. although I'm sure they've got some smart people working on it at Google.
The equivalent is a crawl depth limit, which could be a hard limit (dumb) or a function of page-rank (smart) and the trustworthiness of the inbound link (smarter) and also the data quality & diversity of the traversed pages (best).
There are very good reasons why Googlebot seems to hit you from one IP at a time -- it's a long-running thread that is making all sorts of decisions about your site as it crawls.
Tackling the data quality & diversity of the traversed pages (best):
Producing English text with the 40 bits, by driving a generative grammar or a markov/travesty generator, would make it harder for Google to detect that the pages are auto-generated. It's unlikely to infer the function f(URL) -> text (or even to attempt it), but would limit the recursion for the other reasons you mention.
(guessing) sites like hackernews are indexed primarily by recursion (few direct inbound links to specific stories).
Correct. Notice that it is difficult to find old HN comments on Google, since after a while there are no short paths from the home page to them. In practice & all else being equal (quality, length, spamminess, speed, age, uniqueness, PR, etc), the maximum depth a page can afford to have is about 6 or 7.
Please log requests from the Google and Microsoft bots and let us know how long it takes the respective bots to figure out that every page is the same :-)
Looks like it just hit a bunch of links from the first page.
Maybe I've had too much sugar today, but that made me laugh out loud.
Do all the URLs leave snarky messages?
UFO :D