News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by malf on Friday March 26, 2021 @11:53AM (#26593550)
What do you think this is used for?

https://developers.google.com/search/docs/advanced/crawling/...

[ Parent | on HN ]

There are no comments on this comment yet.