Re:Analyzing One Million Robots.txt Files
by db48x on Monday December 25, 2017 @10:46AM
(#16004488)
It seems pretty obvious: copyright restricts distribution, so they hide pages that the apparent copyright holder apparently doesn't want distributed.
[ Parent
| on HN ]
There are no comments on this comment yet.