News for hackers. Stuff that matters.
Re:Twitter just updated its robots.txt to exclude all scrapers by chc on Wednesday July 15, 2015 @08:14PM (#9894887)
Except don't they have a special deal with Google to use the firehose?
Re:Twitter just updated its robots.txt to exclude all scrapers by guiambros on Wednesday July 15, 2015 @08:34PM (#9894933)
IMO, that's exactly the reason. Before search engines could scrape the data and load the content for free. Now they'll need to reach firehose data agreements.
Re:Twitter just updated its robots.txt to exclude all scrapers by Paulods on Thursday July 16, 2015 @04:38AM (#9896014)
Take a look at the robots file without the "www" subdomain. Its likely to prevent content duplication/push for only the non "www" url to appear on search engines.

https://twitter.com/robots.txt

Re:Twitter just updated its robots.txt to exclude all scrapers by guiambros on Thursday July 16, 2015 @06:36PM (#9900303)
Uh... Excellent point. Seems the entire speculation is/was unfounded.