News for hackers. Stuff that matters.
Re:Google Bot Attempts to Crawl Shortest URLs First by esryl on Saturday June 25, 2011 @10:56AM (#2695765)
The site adheres to a strict url structure. /state/city/id/schoolname - entering from the homepage, the only way to crawl the site 1 level at a time would be crawling the shortest urls first. this structure is also emphasised in the breadcrumbs on every page, the shortest urls are also the ones with the most internal links.

why would you crawl the site in any other way?

Re:Google Bot Attempts to Crawl Shortest URLs First by foxhop on Saturday June 25, 2011 @11:13AM (#2695801)
If you look at a particular city page you will notice that the cities are in alphabetical order, however google bot still crawls by length of url...
Re:Google Bot Attempts to Crawl Shortest URLs First by underdown on Saturday June 25, 2011 @02:56PM (#2696281)
"why would you crawl the site any other way?"

I could see crawling pages most likely to have changed first as those pages would most likely lead to fresh content.