Re:Twitter just updated its robots.txt to exclude all scrapers
by kenkasan on Wednesday July 15, 2015 @07:28PM
(#9894669)
so what does that mean?
[ Parent
| on HN ]
Re:Twitter just updated its robots.txt to exclude all scrapers
by minimaxir on Wednesday July 15, 2015 @07:32PM
(#9894682)
Absolutely nothing from Twitter should be appearing in search engines.
Re:Twitter just updated its robots.txt to exclude all scrapers
by Paulods on Thursday July 16, 2015 @04:37AM
(#9896009)
Actually i believe its only from the "www" subdomain. Take a look at the robots without the "www"
https://www.twitter.com/robots.txt https://twitter.com/robots.txt
This is likely just to prevent content duplication/nudge users to visit without the "www".
Re:Twitter just updated its robots.txt to exclude all scrapers
by thenextcorner on Thursday July 16, 2015 @03:10PM
(#9899283)
yep makes a huge difference... as explained here as well http://webmarketingschool.com/no-twitter-did-not-just-de-ind...
Re:Twitter just updated its robots.txt to exclude all scrapers
by sp332 on Thursday July 16, 2015 @03:23PM
(#9899346)
You should never use Google's estimate as a real estimate, especially once it gets past 100. There are better ways to move content, like 301 redirects or rel=canonical.
Re:Twitter just updated its robots.txt to exclude all scrapers
by teaneedz on Wednesday July 15, 2015 @08:22PM
(#9894904)
So only firehose agreements get data? What's the impact to general SEO or 3rd parties? Doesn't sound good.
Re:Twitter just updated its robots.txt to exclude all scrapers
by chc on Wednesday July 15, 2015 @08:14PM
(#9894887)
Except don't they have a special deal with Google to use the firehose?
Re:Twitter just updated its robots.txt to exclude all scrapers
by guiambros on Wednesday July 15, 2015 @08:34PM
(#9894933)
IMO, that's exactly the reason. Before search engines could scrape the data and load the content for free. Now they'll need to reach firehose data agreements.