What you are describing is exactly what puppeteer uses internally, you might want to explore their code base.

While not currently "easy", there exists the Chrome Devtools Protocol.[0] I'm not aware of a CLI utility that communicates with it, but it wouldn't be impossible to make one that fulfills what you're looking for. A second tool could then act as a REST proxy, if calling the commands via curl is really your jam.

I think you've given my weekend some purpose. Lemme see what I can pull together...

[0] https://chromedevtools.github.io/devtools-protocol/

Yes, that might work. Maybe an even better approach is to use chromium-chromedriver.

I just got it working like this:

    apt install chromium-chromedriver
    chromedriver

This seems to create a service that listens on port 9515 for standardized commands to remote control a chromium instance. The commands seem to be specified by the W3C:

https://www.w3.org/TR/webdriver/

I got it to open a browser with this curl command:

    curl  -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }'  http://localhost:9515/session

I have not yet figured out how to send javascript commands though.

You can do this with Selenium pretty easily, and low level webdrivers support it too.

I would give low level webdrivers a go. But so far I have not even figured out how to install one for Chromium on Debian.

    apt install chromedriver

gives me:

    Package 'chromedriver' has no installation candidate

There is something called "chromium-chromedriver".

Let me try that ... one moment ...

Ok. So I start it via:

    apt install chromium-chromedriver

Now according to the docs, this should create a browser:

    curl  -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }'  http://localhost:9515/session

Ha! It works!

So chromedriver might be a solution!

I've written something[1] that can basically do this, though it's non-interactive.

The CLI interface is somewhat incomplete as-of-now, but it'd be fairly easy to add more comprehensive tools.

https://github.com/fake-name/ChromeController

What would you do with this solution that can’t already be done with headless browsers, apart from looking at it?

It’s very much like that already: write script, send to browser, let it do it’s thing, run javascript code if you want and get the final renderend HTML and console output.

    What would you do with this solution

Automate the browser

    that can’t already be done

I did not say it can't be done now. But I don't want the overhead of pupeteer, selenium plus the client libraries to control them. These things are fricking monsters. And they will go out of fashion at some point. Simple javascript commands will not.

    with headless browsers

I don't want headless. I just want to automate.

But I don't want the overhead of pupeteer

I don't understand this objection. Puppeteer is the client library that you want, and it's not particularly massive. It's the plumbing to handle all of the stuff around starting and connecting to browsers, providing a tidy API to abstract the stuff that's annoying to use directly.

In fact, you can do almost anything you want to using the existing interface that Chrome has built in:

  chromium --remote-debugging-port=9222

You can connect to it using e.g. the WS command line tool using the URL it spits out:

  ws ws://127.0.0.1:9222/devtools/browser/81b9b178-ece0-4953-8ea0-bce6ac31d89c

Then you can make it do things:

  {"id": 1, "method": "Target.createTarget", "params": {"url": "http://www.google.com"}}

It does what you want, over a web socket interface rather than HTTP.

That looks interesting.

Where does that ws tool come from? Is it this one? https://packages.debian.org/stretch/node-ws

How do you send javascript commands over that protocol?

You can run puppeteer with any chromium binary, including non-headless mode.

You can also just start chromium with debugging active `--remote-debugging-port=9222` and then connect to that port and send any dev console commands.

How do you send dev console commands to that port?

It uses the devtools protocol, see here https://chromedevtools.github.io/devtools-protocol/

Looks like it's a REST protocol.

You have to connect to a tab and then call e.g.

    Runtime.evaluate {expression: "console.log('test')"}

Also, you can apparently connect the puppeteer client to a running chrome instance [1]. That way you can use the nice and statically typed puppeteer api and you should have pretty much zero overhead.

[1] https://github.com/GoogleChrome/puppeteer/issues/238

With ‘native messaging,’ you can have your program communicate with your extension, so the extension could then do everything that's available to it via the API.

Won't be surprised if an extension like this already exists.