News for hackers. Stuff that matters.
Re:Does Google crawl dynamic content? by anarchitect on Tuesday December 08, 2015 @08:15AM (#10696105)
We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages).

This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.

There are no comments on this comment yet.