News for hackers. Stuff that matters.
Re:Robots.txt by a5seo on Monday February 11, 2013 @03:06PM (#5202910)
Not to mention helping sites avoid crawl suicide like massively duplicating content across sites, implementing faceted nav and pagination wrong, putting core content in AJAX, generating thin pages, making Google guess which page is the canonical one, and product managers who think "hey let's republish this database of millions of items on our site with generate millions of pages."

There are no comments on this comment yet.