Re:Google has added DuckDuckGo as a search engine option for...
by _bxg1 on Thursday March 14, 2019 @10:54AM
(#19389747)
Headless Chrome is available and widely-used, and commonly you can get around the JS thing by simply waiting a few seconds before scraping. I'd assume the crawling itself isn't the hard part (aside from maybe just the raw compute time it takes).
For one thing, loads of sites load all their content via Ajax, so at a minimum you're gonna need a browser engine as the base of your crawler...