News for hackers. Stuff that matters.
Re:Google Bot Attempts to Crawl Shortest URLs First by arn on Saturday June 25, 2011 @11:07AM (#2695782)
I noticed this behavior also when I was following Googlebot's crawl of my old pages after I had done a redesign.

it's not because of sitemap or because of url structure or because of dynamic content.

Mine were blog articles in the same format. This is how it was crawled:

sitename.com/year/mo/day/stub

sitename.com/year/mo/day/stub-one

sitename.com/year/mo/day/stub-one-two

sitename.com/year/mo/day/stub-one-two-three

sitename.com/year/mo/day/stub-one-two-three-four

There are no comments on this comment yet.