Re:Only Google is really allowed to crawl the web
by aabhay on Friday March 26, 2021 @08:20PM
(#26598559)
Fine, the content is free. But if your crawlers want access to the content, then pay! Simple as that.
Re:Only Google is really allowed to crawl the web
by rezonant on Friday March 26, 2021 @10:24PM
(#26599304)
Will it be a flat fee, so that I, a lowly one-man crawler developer will not be able to afford it? Will it be that only Google can afford it, thus making their monopoly position even stronger?
Is there a Wikipedia crawling "welfare" program if I'm not a trillion dollar mega company?
Re:Only Google is really allowed to crawl the web
by dannyw on Friday March 26, 2021 @11:08PM
(#26599565)
Actually, yes! The current API is free. The new Enterprise API is paid.
Re:Only Google is really allowed to crawl the web
by aabhay on Saturday March 27, 2021 @12:07PM
(#26603290)
Sure! Apply to become a crawler. And if you meet certain criteria and your crawlers don’t exceed a quota then have at it. The key is not to make it technically challenging, but to erect a legal barrier.
They may be able to charge for bandwidth (if you want to use a Wikipedia image, you can use Wikipedia's enterprise CDN instead of their own), but their licensing allows me to rehost content as long as I follow the attribution & sublicensing terms.
Google has no problem operating their own CDNs, so I find it unlikely that Wikipedia will be able to monetize Google search results in such a manner as you described.
Disclaimer: I work for Google; opinions are my own.