Re:Twitter rewrites developer policy to better support resea...
by CaptArmchair on Wednesday March 11, 2020 @06:25AM
(#22544581)
You're not wrong. Library of Congress started archiving every public tweet in 2010 but abandoned the effort in 2017 mainly because of the sheer volume.
I think it might be feasible to scrape a subset: a single account or a few accounts at most. But even then, you're still fighting problems ranging from wonky HTML generated through JavaScript towards sudden, unannounced changes to the DOM structure. It's not a managed API after all.
https://www.npr.org/sections/thetwo-way/2017/12/26/573609499...
I think it might be feasible to scrape a subset: a single account or a few accounts at most. But even then, you're still fighting problems ranging from wonky HTML generated through JavaScript towards sudden, unannounced changes to the DOM structure. It's not a managed API after all.