Scraping web sites which dynamically load data (like Twitter)

Re:Scraping web sites which dynamically load data (like Twit... by wslh on Monday July 29, 2013 @08:45PM (#6124802)

For people who don't have much experience in web scraping, it's important to note that there is no a better/worse approach. Everything depends on the specific site.

I don't see (2) as a simpler approach that scraping from the browser rendered page: the information retrieved between the browser and the server can be obfuscated and will require too much effort to analyze the protocol between them. You mention the Sockets module but if you are using HTTPS request the information at that level will be encrypted.

I really encourage people to explore the Data Big Bang site since offers a lot of different approaches for web scraping. Just look at the resources at the end of the http://blog.databigbang.com/web-scraping-ajax-and-javascript... article.

Why use Google Chrome (or Firefox)? One of the reasons is that you are sure that it is updated very often and their performance is great.

[ Parent | on HN ]

Re:Scraping web sites which dynamically load data (like Twit... by malandrew on Tuesday July 30, 2013 @02:50AM (#6125660)

I wasn't suggested sniffing traffic. With WebSockets, it's very hard to do in an automated way. I was suggesting looking at the Models and Collections in a web app via de-obfuscation and a little reverse engineering. Injecting JavaScript into a page via PhantomJS is a great way to control a web app.

[ Parent | on HN ]