News for hackers. Stuff that matters.
Re:Twitter just updated its robots.txt to exclude all scrapers by Paulods on Thursday July 16, 2015 @04:37AM (#9896009)
Actually i believe its only from the "www" subdomain. Take a look at the robots without the "www"

https://www.twitter.com/robots.txt https://twitter.com/robots.txt

This is likely just to prevent content duplication/nudge users to visit without the "www".

Re:Twitter just updated its robots.txt to exclude all scrapers by thenextcorner on Thursday July 16, 2015 @03:10PM (#9899283)
yep makes a huge difference... as explained here as well http://webmarketingschool.com/no-twitter-did-not-just-de-ind...
Re:Twitter just updated its robots.txt to exclude all scrapers by sp332 on Thursday July 16, 2015 @03:23PM (#9899346)
You should never use Google's estimate as a real estimate, especially once it gets past 100. There are better ways to move content, like 301 redirects or rel=canonical.