Re:Does Google crawl dynamic content?
by anarchitect on Tuesday December 08, 2015 @08:15AM
(#10696105)
We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages).
This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.
This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.