Re:Chinese firm Huawei controls net filter praised by PM
by auxbuss on Friday July 26, 2013 @09:26AM
(#6108057)
They also send a crawler to any urls that they've not "registered" before (by IP). About 30 secs after navigating to a new page their bot will crawl a visited page. robots.txt is not honoured.
Something to consider when testing, or using, "hidden" sites and pages.
Something to consider when testing, or using, "hidden" sites and pages.