Re:Google Bot Attempts to Crawl Shortest URLs First
by georgemcbay on Saturday June 25, 2011 @04:24PM
(#2696469)
I suspect you're on the right track with your first guess.
Most people posting here are looking for some sort of deep meaning in this when IMO it is more likely just due to a localized side-effect of doing something such as storing the urls in a trie-like structure and then iterating over it breadth-first.
Most people posting here are looking for some sort of deep meaning in this when IMO it is more likely just due to a localized side-effect of doing something such as storing the urls in a trie-like structure and then iterating over it breadth-first.