News for hackers. Stuff that matters.
Re:Web Scraping with Electron by aboutruby on Saturday March 09, 2019 @09:32AM (#19346258)
Interesting but seems less powerful than my current setup:

- I have mitmproxy to capture the traffic / manipulate the traffic

- I have Chrome opened with Selenium/Capybara/chromedriver and using mitmproxy

- I then browse to the target pages, it records the selected requests and the selected responses

- It then replays the requests until they fail (with a delay)

I highly recommend mitmproxy, it's extremely powerful: capture traffic, send responses without hitting the server, block/hang requests, modify responses, modify requests/responses headers.

Then higher level interfaces can be built on top, Selenium allows you to load Chrome extensions and execute Javascript on any page for instance. You can also manage many tabs at the same time.

I could make a blog post/demo if people are interested

Re:Web Scraping with Electron by kuhhk on Saturday March 09, 2019 @09:54AM (#19346387)
A blog article would be nice. Sounds interesting, but I’m having a hard-time understanding. If it’s replaying requests, how do you get it to do things like go to the next pagination and click on all of the next paginated results?
Re:Web Scraping with Electron by aboutruby on Saturday March 09, 2019 @10:07AM (#19346457)
In my case I can't do the pagination automatically so I have to fetch the pages myself to then have them replayed.

In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API).

You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows.

The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect).

Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that.

Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev (that I will turn into a blog).

Re:Web Scraping with Electron by marvel_boy on Saturday March 09, 2019 @09:46AM (#19346343)
>I could make a blog post/demo if people are interested

yes, please !