News for hackers. Stuff that matters.
Re:Google Bot Attempts to Crawl Shortest URLs First by christianwilde on Saturday June 25, 2011 @02:02PM (#2696158)
I imagine that in these specific case is because longer URLs represent deeper pages on the site that are less "important" (in terms of internal incoming links and pagerank) than the shorter ones. It doesn't seem logical that google order the URLs by length and then crawl them in that order; probably the URL length can be a factor that the bot takes into account, but not the only one in the manner this article suggest :)

Anyway, good point, that deserves more testing to extract some conclussions

There are no comments on this comment yet.