News for hackers. Stuff that matters.
Re:Is DuckDuckGo, DuckDuckDone? by sct202 on Thursday May 26, 2022 @12:07PM (#31520058)
A lot of the internet is behind semi-walled gardens and GoogleBot and BingBot have preferential access.

All the large social sites, Cloudflare, and other CDNs will just stop serving unknown bots after awhile, which is why there are spidering/scraping services that cycle thru IP addresses. See: https://news.ycombinator.com/item?id=29417061

Re:Is DuckDuckGo, DuckDuckDone? by bombcar on Thursday May 26, 2022 @12:12PM (#31520120)
Sounds like Cloudflare has enough information to become a search engine itself, if it ever wanted to ...