News for hackers. Stuff that matters.
Re:Analyzing One Million Robots.txt Files by zorpner on Sunday December 24, 2017 @02:49PM (#16001009)
The next steps towards standardization began when Google, Yahoo, and Microsoft came together to define and support the sitemap protocol in 2006. Then in 2007, they announced that all three of them would support the Sitemap directive in robots.txt files. And yes, that important piece of internet history from the blog of a formerly 125 Billion dollar company now only exists because it was archived by Archive.org.

The Internet Archive (archive.org) is currently running their end-of-year donation drive, if you value the work they do it's a good time to donate: https://archive.org/donate/

(and on the topic of robots.txt, it sounds like they're moving in the direction of disallowing people from using them indiscriminately to block access to valuable archival materials: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... )

Re:Analyzing One Million Robots.txt Files by stordoff on Sunday December 24, 2017 @07:57PM (#16002102)
Have the IA ever discussed why they retroactively apply robots.txt? I can see the rational (though don't necessarily think it is the best idea given the IA's goals) for respecting it at crawl time, but applying it retroactively always felt unnecessary to me.
Re:Analyzing One Million Robots.txt Files by db48x on Monday December 25, 2017 @10:46AM (#16004488)
It seems pretty obvious: copyright restricts distribution, so they hide pages that the apparent copyright holder apparently doesn't want distributed.