News for hackers. Stuff that matters.
Re:Google Bot Attempts to Crawl Shortest URLs First by JonnieCache on Saturday June 25, 2011 @11:52AM (#2695884)
There is probably some highly non-obvious reason that sorting your queue of URLs by length is optimal, which was arrived at after a lot of modelling and testing.

We're unlikely to ever know the answer unless someone from google explains it to us.

Re:Google Bot Attempts to Crawl Shortest URLs First by JonnieCache on Saturday June 25, 2011 @01:57PM (#2696144)
Thinking about it more, its probably just a breadth-first search. Duh.
Re:Google Bot Attempts to Crawl Shortest URLs First by frisco on Saturday June 25, 2011 @03:43PM (#2696385)
Breadth-first search means crawling all of the links on the page and adding all of the links on the child pages to the queue at once rather than drilling down on one link-path first before moving to the other links on the first page. There's probably some highly non-obvious reason for crawling by url length.
Re:Google Bot Attempts to Crawl Shortest URLs First by xyzzyz on Sunday June 26, 2011 @06:12AM (#2697766)
The thing is, the deeper you are, the longer are the urls, so if you do a breadth first search, you are more likely to visit short urls first.