Creating an open-source solution to the headaches of headless browsers

Re:Creating an open-source solution to the headaches of head... by JustARandomGuy on Monday November 25, 2019 @07:07PM (#21634327)

Since you're reading this, a feature request: I would love it if you could put up a REST endpoint for extracting all the images (example code here [1]) on a web page, and more endpoints for extracting all the links, script addresses, etc.

I was trying to do that on Browserless but couldn't get the final file download to work (I adapted Stack Overflow code linked below to put all the web pages images into a ZIP file and download that) - presently I'm running this on a Google Cloud Function, which is working but I'd rather outsource it to you, especially since the function chokes on large web pages (possibly it needs more RAM than the 2GB limit currently available in GCF?).

[1] https://stackoverflow.com/a/52542490

[ Parent | on HN ]

Re:Creating an open-source solution to the headaches of head... by mrskitch on Tuesday November 26, 2019 @03:15PM (#21642206)

Just to follow up, you can use our /scrape API to do this:

curl -X POST \ https://chrome.browserless.io/scrape \ -H 'Content-Type: application/json' \ -d '{ "url": "https://reddit.com/", "elements": [{ "selector": "img" }] }'

This will get all the <img> tags on a page and return their attributes (which includes their sources). If you wanted to do scripts as well, just add a new object to the elements array with the "selector" of "script".

[ Parent | on HN ]

Re:Creating an open-source solution to the headaches of head... by mrskitch on Monday November 25, 2019 @07:21PM (#21634410)

Noted ;)

[ Parent | on HN ]