News for hackers. Stuff that matters.

A comment on Scraping web sites which dynamically load data (like Twitter). Parent comment

Re:Scraping web sites which dynamically load data (like Twit... by nemothekid on Monday July 29, 2013 @07:08PM (#6124451)
(2) is also much less resource intensive as well. Who knows how much JS on may end up executing on Twitter's homepage.
[ Parent | on HN ]
Re:Scraping web sites which dynamically load data (like Twit... by wslh on Monday July 29, 2013 @08:58PM (#6124834)
Benchmark your personal time developing a Twitter web scraper similar to this one to the time to follow the (2) approach. The code of the article was done in a few hours while the (2) approach would require further analysis and risks (analyzing protocols can be very tricky). Google Chrome and Firefox are very fast and one developer day may cost more than one PC.
[ Parent | on HN ]
Re:Scraping web sites which dynamically load data (like Twit... by nemothekid on Monday July 29, 2013 @10:36PM (#6125094)
Well at the end of the day, the application really matters. In most cases I find just sniffing the XHR requests with Chrome's Developer Tools to be the most productive.
[ Parent | on HN ]