Re:Web Scraping a Javascript Heavy Website: Keeping Things S...
by PMan74 on Thursday August 29, 2013 @05:11AM
(#6294957)
Wow I thought mitmproxy looked rough until I saw tcpdump/ngrep/awk? They both work of course but neither look especially easy to use.
We've been using http://www.charlesproxy.com/ for years, great tool (cheap albeit not free)
Below is a simple, _lightweight_ ngrep solution. RE means a regular expression. This only saves packets with the RE you specify and does not save full packets, only the HTTP headers. 1024 is an arbitrary size to get all HTTP headers; adjust to taste. tcpdump is there only because ngrep does not work well with PPPoE. If you don't use PPPoE you don't need to include tcpdump.
To dump your results, try And here's a little script to make URL's from your pcap file. unvis just decodes URL's from the specs in RFC's 1808 and 1866. It assumes http:// URL's (no ftp://). The awk script ensures all URL's (not just consecutive ones) are unique. It's trivial to dump HTTP. You can feed this to netcat (using sed to modify the HTML to your liking), then open the result in your browser. Whatever you are aiming to do (I'm still not exactly sure - can you give an example?), I reckon it can be automated without Python and heaps of libraries.