Re:Google Bot Attempts to Crawl Shortest URLs First
by _grrr on Saturday June 25, 2011 @11:57AM
(#2695900)
A poor mans PageRank algorithm, assuming nothing else, would assign a higher PageRank to shorter site links on a page. Presumably the crawler visits pages with a higher PR first.
[ Parent
| on HN ]
Re:Google Bot Attempts to Crawl Shortest URLs First
by TuxPirate on Saturday June 25, 2011 @12:37PM
(#2695980)
Pagerank determine crawls rate amongst other things. I find it is also likely that short URLs (especially in the case of a directory-type site) are seen first by the spider and that this order is respected by the crawler (FIFO).
You can also ask in #seo on irc.freenode.net I know there are knowledgable SEO people in there who might be able to provide you with a decent answer.