News for hackers. Stuff that matters.
Re:Chinese firm Huawei controls net filter praised by PM by auxbuss on Friday July 26, 2013 @09:26AM (#6108057)
They also send a crawler to any urls that they've not "registered" before (by IP). About 30 secs after navigating to a new page their bot will crawl a visited page. robots.txt is not honoured.

Something to consider when testing, or using, "hidden" sites and pages.

There are no comments on this comment yet.