Re:Twitter just updated its robots.txt to exclude all scrapers
by thenextcorner on Thursday July 16, 2015 @03:10PM
(#9899283)
yep makes a huge difference... as explained here as well http://webmarketingschool.com/no-twitter-did-not-just-de-ind...
https://www.twitter.com/robots.txt https://twitter.com/robots.txt
This is likely just to prevent content duplication/nudge users to visit without the "www".