News for hackers. Stuff that matters.

A comment on Twitter just updated its robots.txt to exclude all scrapers. Parent comment

Re:Twitter just updated its robots.txt to exclude all scrapers by sp332 on Thursday July 16, 2015 @10:33PM (#9901041)
They could at least have allowed the Internet Archive in the robots.txt, since the way things stand all www.twitter.com links will be unavailable from the Wayback Machine. That will obviously be a huge loss to researchers.
[ Parent | on HN ]
Re:Twitter just updated its robots.txt to exclude all scrapers by sp332 on Friday July 17, 2015 @11:49AM (#9903301)
(Update: the Wayback Machine will be fine, using the twitter.com/robots.txt)
[ Parent | on HN ]