Re:Google Bot Attempts to Crawl Shortest URLs First
by JonnieCache on Saturday June 25, 2011 @01:57PM
(#2696144)
Thinking about it more, its probably just a breadth-first search. Duh.
Re:Google Bot Attempts to Crawl Shortest URLs First
by frisco on Saturday June 25, 2011 @03:43PM
(#2696385)
Breadth-first search means crawling all of the links on the page and adding all of the links on the child pages to the queue at once rather than drilling down on one link-path first before moving to the other links on the first page. There's probably some highly non-obvious reason for crawling by url length.
We're unlikely to ever know the answer unless someone from google explains it to us.