Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Google competitor DuckDuckGo says it's getting shut out. Parent comment
Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
There are no comments on this comment yet.
Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?