News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by foobar33333 on Friday March 26, 2021 @05:38PM (#26597274)
On smaller sites, nothing usually. But on bigger sites you will be blocked. You will probably be blocked even if you do follow robots.txt
[ Parent | on HN ]

There are no comments on this comment yet.