Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Web Scraping with Electron. Parent comment
In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API).
You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows.
The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect).
Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that.
Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev (that I will turn into a blog).
There are no comments on this comment yet.
In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API).
You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows.
The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect).
Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that.
Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev (that I will turn into a blog).