News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by rezonant on Friday March 26, 2021 @10:24PM (#26599304)
Will it be a flat fee, so that I, a lowly one-man crawler developer will not be able to afford it? Will it be that only Google can afford it, thus making their monopoly position even stronger?

Is there a Wikipedia crawling "welfare" program if I'm not a trillion dollar mega company?

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by dannyw on Friday March 26, 2021 @11:08PM (#26599565)
Actually, yes! The current API is free. The new Enterprise API is paid.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by aabhay on Saturday March 27, 2021 @12:07PM (#26603290)
Sure! Apply to become a crawler. And if you meet certain criteria and your crawlers don’t exceed a quota then have at it. The key is not to make it technically challenging, but to erect a legal barrier.
[ Parent | on HN ]