Web Scraping with Electron

Re:Web Scraping with Electron by dlkinney on Saturday March 09, 2019 @08:16AM (#19345964)

While not currently "easy", there exists the Chrome Devtools Protocol.[0] I'm not aware of a CLI utility that communicates with it, but it wouldn't be impossible to make one that fulfills what you're looking for. A second tool could then act as a REST proxy, if calling the commands via curl is really your jam.

I think you've given my weekend some purpose. Lemme see what I can pull together...

[0] https://chromedevtools.github.io/devtools-protocol/

[ Parent | on HN ]

Re:Web Scraping with Electron by TicklishTiger on Saturday March 09, 2019 @08:23AM (#19345997)

Yes, that might work. Maybe an even better approach is to use chromium-chromedriver.

I just got it working like this:

    apt install chromium-chromedriver
    chromedriver

This seems to create a service that listens on port 9515 for standardized commands to remote control a chromium instance. The commands seem to be specified by the W3C:

https://www.w3.org/TR/webdriver/

I got it to open a browser with this curl command:

    curl  -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }'  http://localhost:9515/session

I have not yet figured out how to send javascript commands though.

[ Parent | on HN ]