Re:Google Bot Attempts to Crawl Shortest URLs First
by christianwilde on Saturday June 25, 2011 @02:02PM
(#2696158)
I imagine that in these specific case is because longer URLs represent deeper pages on the site that are less "important" (in terms of internal incoming links and pagerank) than the shorter ones. It doesn't seem logical that google order the URLs by length and then crawl them in that order; probably the URL length can be a factor that the bot takes into account, but not the only one in the manner this article suggest :)
Anyway, good point, that deserves more testing to extract some conclussions
Anyway, good point, that deserves more testing to extract some conclussions