News for hackers. Stuff that matters.
Re:Analyzing One Million Robots.txt Files by db48x on Monday December 25, 2017 @10:46AM (#16004488)
It seems pretty obvious: copyright restricts distribution, so they hide pages that the apparent copyright holder apparently doesn't want distributed.

There are no comments on this comment yet.