Re:Observations running 2M headless sessions
by RyanShook on Monday June 04, 2018 @09:04PM
(#17234132)
I’d like to know, what are people on HN using headless browsers for?
[ Parent
| on HN ]
Re:Observations running 2M headless sessions
by mrskitch on Monday June 04, 2018 @09:22PM
(#17234211)
You’d actually be surprised by the applications, I definitely was. Some I’ve seen:
- Screenshots
- PDF generation
- Functional tests
- Regression tests
- Scraping SPAs
- Crazy stuff like printing user-generated content for personalization products..
Re:Observations running 2M headless sessions
by leetbulb on Monday June 04, 2018 @09:49PM
(#17234336)
Among some of these, I've also built an internal microservice for analyzing URL redirect chains. Input a URL and it will provide me with all of the redirects it goes through and information for each one including method (html meta, 301, etc), timing, etc.
I’ve done some hacky work redirect analysis but just the final url. Thought of open sourcing?
Re:Observations running 2M headless sessions
by skinnymuch on Tuesday June 05, 2018 @10:41AM
(#17238052)
Awesome! What did you use to go through the redirects? I’ve had issues with some browser automation only giving the final url.
Re:Observations running 2M headless sessions
by heipei on Tuesday June 05, 2018 @02:35AM
(#17235438)
I'm using Chrome Headless to analyze user-submitted pages on https://urlscan.io
I will save a screenshot, network request info, request content (for JS/CSS/HTML), cookies, log messages, JavaScript global variables etc. Annotate and correlate it all and you have a nice summary of what a certain website "does".
Re:Observations running 2M headless sessions
by stcredzero on Monday June 04, 2018 @09:49PM
(#17234335)
I am using them to run node-electron hosted WebRTC proxies, to enable UDP-like networking for a real time browser game.
Re:Observations running 2M headless sessions
by williamxd3 on Tuesday June 05, 2018 @03:58AM
(#17235734)
interacting with an app which doesn't provides a public API and has too much frontend code to be worth reverse engineering, I just hack into the app's webpack bundled code and use it
Re:Observations running 2M headless sessions
by greenleafjacob on Tuesday June 05, 2018 @01:58AM
(#17235298)
I'm using splash [1] for scraping websites for personal automation purposes.
I scrape lots of data from services while logged in. Scraping is done with curl, but every so often cookies expire, so I use selenium to login, save cookies, then go back to curl.
Re:Observations running 2M headless sessions
by smallbigfish on Monday June 04, 2018 @11:56PM
(#17234869)
Isn't curl fit to also do the login part and save the cookies?
Re:Observations running 2M headless sessions
by ikeboy on Tuesday June 05, 2018 @12:06AM
(#17234907)
Tons of stuff get loaded during login, hard to go through each one and code the proper urls and POST params etc, quicker to write login code.
Re:Observations running 2M headless sessions
by mskullcap on Tuesday June 05, 2018 @09:08AM
(#17237248)
PDF generation for us. Had been relying on phantomjs for years, but it has been a bear. Attempted to write something with PDFKit to avoid the overhead of hosting headless chrome, but the effort was too great for our small team. Using chrome we are able to build high fidelity reports with the same web components we use in the application's UI.
Re:Observations running 2M headless sessions
by tnolet on Tuesday June 05, 2018 @05:04AM
(#17236011)
I'm using Puppeteer for website / transaction monitoring at https://checklyhq.com. People script their own checks and we run them for you on a cron like schedule and alert when things go off the rails.
Re:Observations running 2M headless sessions
by fake-name on Monday June 04, 2018 @10:11PM
(#17234427)
Personally? Dealing with cloudflare and other bullshit WAFs that break the internet.
Re:Observations running 2M headless sessions
by manquer on Monday June 04, 2018 @11:22PM
(#17234733)
Recording of webrtc video conferencing sessions , pdf generation, spa test suites
https://www.prerender.cloud/ - server-side rendering API for single-page apps
Headless browsers are the de-facto standard for scrapping websites which have some protection layer. Ad-fraud is also a thing.
Re:Observations running 2M headless sessions
by deejbee on Tuesday June 05, 2018 @07:49AM
(#17236764)
Automating logging into and data capture for financial/banking sites, building a new finance aggregation service.
Re:Observations running 2M headless sessions
by PetahNZ on Monday June 04, 2018 @10:29PM
(#17234492)
Rendering high res vectors (SVG)
Re:Observations running 2M headless sessions
by orliesaurus on Tuesday June 05, 2018 @03:46AM
(#17235680)
Built an DOM 2 png image api (hivis.io)
Re:Observations running 2M headless sessions
by stevenjohns on Monday June 04, 2018 @09:05PM
(#17234138)
Personally, for testing.