News for hackers. Stuff that matters.
Re:Analyzing One Million Robots.txt Files by stordoff on Sunday December 24, 2017 @07:57PM (#16002102)
Have the IA ever discussed why they retroactively apply robots.txt? I can see the rational (though don't necessarily think it is the best idea given the IA's goals) for respecting it at crawl time, but applying it retroactively always felt unnecessary to me.
Re:Analyzing One Million Robots.txt Files by db48x on Monday December 25, 2017 @10:46AM (#16004488)
It seems pretty obvious: copyright restricts distribution, so they hide pages that the apparent copyright holder apparently doesn't want distributed.