Re:Google Bot Attempts to Crawl Shortest URLs First
by personalcompute on Saturday June 25, 2011 @11:36AM
(#2695850)
I did a more scientific analysis of the googlebot requests in the provided log (graph! http://i.imgur.com/uMoUT.png) and it definitely looks like it is taking shortest urls first. Anyone else with a large site want to check as well for further data?
[ Parent
| on HN ]