News for hackers. Stuff that matters.

A comment on Twitter rewrites developer policy to better support research and ‘good’ bots. Parent comment

Re:Twitter rewrites developer policy to better support resea... by godzillabrennus on Tuesday March 10, 2020 @05:16PM (#22539799)
Scraping, while legal now, is still not free. A scrape does not give you the delta between two points in time without a completely new scrape. You cannot economically sustainably scrape the entirety of a large site like Twitter.

If I’m wrong I’d love to learn from someone smarter than me.

Even google doesn’t scrape the entire web every day.

[ Parent | on HN ]
Re:Twitter rewrites developer policy to better support resea... by CaptArmchair on Wednesday March 11, 2020 @06:25AM (#22544581)
You're not wrong. Library of Congress started archiving every public tweet in 2010 but abandoned the effort in 2017 mainly because of the sheer volume.

https://www.npr.org/sections/thetwo-way/2017/12/26/573609499...

I think it might be feasible to scrape a subset: a single account or a few accounts at most. But even then, you're still fighting problems ranging from wonky HTML generated through JavaScript towards sudden, unannounced changes to the DOM structure. It's not a managed API after all.

[ Parent | on HN ]