News for hackers. Stuff that matters.
Re:Web Scraping with Electron by TicklishTiger on Saturday March 09, 2019 @08:09AM (#19345943)

    What would you do with this solution
Automate the browser

    that can’t already be done
I did not say it can't be done now. But I don't want the overhead of pupeteer, selenium plus the client libraries to control them. These things are fricking monsters. And they will go out of fashion at some point. Simple javascript commands will not.

    with headless browsers
I don't want headless. I just want to automate.
Re:Web Scraping with Electron by matthewmacleod on Saturday March 09, 2019 @08:34AM (#19346022)
But I don't want the overhead of pupeteer

I don't understand this objection. Puppeteer is the client library that you want, and it's not particularly massive. It's the plumbing to handle all of the stuff around starting and connecting to browsers, providing a tidy API to abstract the stuff that's annoying to use directly.

In fact, you can do almost anything you want to using the existing interface that Chrome has built in:

  chromium --remote-debugging-port=9222
You can connect to it using e.g. the WS command line tool using the URL it spits out:

  ws ws://127.0.0.1:9222/devtools/browser/81b9b178-ece0-4953-8ea0-bce6ac31d89c
Then you can make it do things:

  {"id": 1, "method": "Target.createTarget", "params": {"url": "http://www.google.com"}}
It does what you want, over a web socket interface rather than HTTP.
Re:Web Scraping with Electron by TicklishTiger on Saturday March 09, 2019 @08:56AM (#19346104)
That looks interesting.

Where does that ws tool come from? Is it this one? https://packages.debian.org/stretch/node-ws

How do you send javascript commands over that protocol?

Re:Web Scraping with Electron by phiresky on Saturday March 09, 2019 @08:19AM (#19345982)
You can run puppeteer with any chromium binary, including non-headless mode.

You can also just start chromium with debugging active `--remote-debugging-port=9222` and then connect to that port and send any dev console commands.

Re:Web Scraping with Electron by TicklishTiger on Saturday March 09, 2019 @08:25AM (#19345999)
How do you send dev console commands to that port?
Re:Web Scraping with Electron by phiresky on Saturday March 09, 2019 @08:38AM (#19346032)
It uses the devtools protocol, see here https://chromedevtools.github.io/devtools-protocol/

Looks like it's a REST protocol.

You have to connect to a tab and then call e.g.

    Runtime.evaluate {expression: "console.log('test')"}
Re:Web Scraping with Electron by phiresky on Saturday March 09, 2019 @08:40AM (#19346039)
Also, you can apparently connect the puppeteer client to a running chrome instance [1]. That way you can use the nice and statically typed puppeteer api and you should have pretty much zero overhead.

[1] https://github.com/GoogleChrome/puppeteer/issues/238