This is approximately how I was thinking of settings thing up as well, so thanks for this!
https://github.com/RealEstateWebTools/property_web_scraper
Will dig into your code to see what I can learn.
Chrome headless should be fastest. And if you're printing to pdf phantom and chrome had the better ability to do that..
I had each sidekiq job spinning up its own process -- and needed to do some X display avoidance.
https://docs.slimerjs.org/master/release-notes-1.0.html
"With Firefox 56 and more, SlimerJS can be trully headless by adding the –headless option on the command line"
So that could be an interesting thing to play with. I've thought about just having my sidekiq jobs spin up a docker container with the browser in it and then just limiting the amount of jobs at once for resource constraints, and now seeing your comment I think I might just give it a shot.
I found browserless.io and the supporting github project over the weekend, looks promising.
Who else is doing this? Is the architecture I just touched on the right way to go about doing this? Or should I have each sidekiq job spin up its own browser container for the scraping job?
Also, chrome vs firefox won't really matter in this instance, but reliability/performance/memory will...any recommendations?