Google says there are now 10^12 unique URLs. I am proud to announce the 2nd trillion.

Re:Google says there are now 10^12 unique URLs. I am proud t... by snprbob86 on Sunday July 27, 2008 @11:32PM (#258681)

I submitted http://ianab.com/trillion/0.html to http://www.google.com/addurl/ and http://search.msn.com.sg/docs/submit.aspx

Please log requests from the Google and Microsoft bots and let us know how long it takes the respective bots to figure out that every page is the same :-)

[ Parent | on HN ]

Re:Google says there are now 10^12 unique URLs. I am proud t... by bdr on Monday July 28, 2008 @01:40AM (#258810)

http://pastebin.com/m54b7d354

Looks like it just hit a bunch of links from the first page.

[ Parent | on HN ]

Re:Google says there are now 10^12 unique URLs. I am proud t... by snprbob86 on Monday July 28, 2008 @03:34AM (#258873)

So it looks like Google grabbed about 40 links before giving up? I wonder what a good "score" is? At first, I'd guess less is better, but too few might be running the risk of throwing out potentially good pages. Too many, and the bot is just wasting effort. The 40 score could vary as well based on parallel conditions assuming many bot instances are sharing a task pool. Be sure to post the Microsoft results if/when they crawl you.

[ Parent | on HN ]

Re:Google says there are now 10^12 unique URLs. I am proud t... by brianr on Monday July 28, 2008 @07:37AM (#258981)

Looks like it hit all of the links on the first page (there are 8x5 boxes = 40) and didn't find anything interesting, so it didn't crawl any deeper. If the second-level pages had more interesting/unique content, I bet it would've kept going.

[ Parent | on HN ]

Re:Google says there are now 10^12 unique URLs. I am proud t... by hhm on Monday July 28, 2008 @09:06AM (#259020)

Thanks a lot for sharing this. Very interesting indeed.

[ Parent | on HN ]