News for hackers. Stuff that matters.
Re:Only Google is really allowed to crawl the web by aabhay on Saturday March 27, 2021 @12:07PM (#26603290)
Sure! Apply to become a crawler. And if you meet certain criteria and your crawlers don’t exceed a quota then have at it. The key is not to make it technically challenging, but to erect a legal barrier.

There are no comments on this comment yet.