News for hackers. Stuff that matters.
Re:Google has added DuckDuckGo as a search engine option for... by _bxg1 on Wednesday March 13, 2019 @04:59PM (#19383709)
How does that work with their privacy stance? Do Yahoo/Bing get to keep and use that search data and it's just anonymized, or does DDG pay to keep it untracked?

Kind of disheartening regardless. I assumed they had their own scrappy, independent tech stack.

Re:Google has added DuckDuckGo as a search engine option for... by londons_explore on Thursday March 14, 2019 @04:44AM (#19387471)
Creating your own search engine in today's world is pretty much impossible.

For one thing, loads of sites load all their content via Ajax, so at a minimum you're gonna need a browser engine as the base of your crawler...

Re:Google has added DuckDuckGo as a search engine option for... by _bxg1 on Thursday March 14, 2019 @10:54AM (#19389747)
Headless Chrome is available and widely-used, and commonly you can get around the JS thing by simply waiting a few seconds before scraping. I'd assume the crawling itself isn't the hard part (aside from maybe just the raw compute time it takes).