Re:Twitter just updated its robots.txt to exclude all scrapers
by chc on Wednesday July 15, 2015 @08:14PM
(#9894887)
Except don't they have a special deal with Google to use the firehose?
[ Parent
| on HN ]
Re:Twitter just updated its robots.txt to exclude all scrapers
by guiambros on Wednesday July 15, 2015 @08:34PM
(#9894933)
IMO, that's exactly the reason. Before search engines could scrape the data and load the content for free. Now they'll need to reach firehose data agreements.