Re:Is DuckDuckGo, DuckDuckDone?
by sct202 on Thursday May 26, 2022 @12:07PM
(#31520058)
A lot of the internet is behind semi-walled gardens and GoogleBot and BingBot have preferential access.
All the large social sites, Cloudflare, and other CDNs will just stop serving unknown bots after awhile, which is why there are spidering/scraping services that cycle thru IP addresses. See: https://news.ycombinator.com/item?id=29417061
All the large social sites, Cloudflare, and other CDNs will just stop serving unknown bots after awhile, which is why there are spidering/scraping services that cycle thru IP addresses. See: https://news.ycombinator.com/item?id=29417061