News for hackers. Stuff that matters.

A comment on Twitter just updated its robots.txt to exclude all scrapers. Parent comment

Re:Twitter just updated its robots.txt to exclude all scrapers by sp332 on Thursday July 16, 2015 @03:46PM (#9899465)
Why would you not use 301 redirects or just rel=canonical?
[ Parent | on HN ]
Re:Twitter just updated its robots.txt to exclude all scrapers by searchmartin on Thursday July 16, 2015 @04:40PM (#9899759)
There must be some platform issue is my best guess...
[ Parent | on HN ]
Re:Twitter just updated its robots.txt to exclude all scrapers by sp332 on Thursday July 16, 2015 @10:33PM (#9901041)
They could at least have allowed the Internet Archive in the robots.txt, since the way things stand all www.twitter.com links will be unavailable from the Wayback Machine. That will obviously be a huge loss to researchers.
[ Parent | on HN ]
Re:Twitter just updated its robots.txt to exclude all scrapers by sp332 on Friday July 17, 2015 @11:49AM (#9903301)
(Update: the Wayback Machine will be fine, using the twitter.com/robots.txt)
[ Parent | on HN ]