Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Google Bot Attempts to Crawl Shortest URLs First.
it's not because of sitemap or because of url structure or because of dynamic content.
Mine were blog articles in the same format. This is how it was crawled:
sitename.com/year/mo/day/stub
sitename.com/year/mo/day/stub-one
sitename.com/year/mo/day/stub-one-two
sitename.com/year/mo/day/stub-one-two-three
sitename.com/year/mo/day/stub-one-two-three-four
There are no comments on this comment yet.
it's not because of sitemap or because of url structure or because of dynamic content.
Mine were blog articles in the same format. This is how it was crawled:
sitename.com/year/mo/day/stub
sitename.com/year/mo/day/stub-one
sitename.com/year/mo/day/stub-one-two
sitename.com/year/mo/day/stub-one-two-three
sitename.com/year/mo/day/stub-one-two-three-four