Re:Twitter just updated its robots.txt to exclude all scrapers
by Paulods on Thursday July 16, 2015 @04:38AM
(#9896014)
Take a look at the robots file without the "www" subdomain. Its likely to prevent content duplication/push for only the non "www" url to appear on search engines.
https://twitter.com/robots.txt