Re:Only Google is really allowed to crawl the web
by tylerhou on Friday March 26, 2021 @03:36PM
(#26596040)
Unlikely that Wikipedia will be able to charge for content, seeing as all of their content is CC-BY-SA licensed. https://en.wikipedia.org/wiki/Wikipedia:Licensing_update
They may be able to charge for bandwidth (if you want to use a Wikipedia image, you can use Wikipedia's enterprise CDN instead of their own), but their licensing allows me to rehost content as long as I follow the attribution & sublicensing terms.
Google has no problem operating their own CDNs, so I find it unlikely that Wikipedia will be able to monetize Google search results in such a manner as you described.
Disclaimer: I work for Google; opinions are my own.
Re:Only Google is really allowed to crawl the web
by aabhay on Friday March 26, 2021 @08:20PM
(#26598559)
Fine, the content is free. But if your crawlers want access to the content, then pay! Simple as that.
Re:Only Google is really allowed to crawl the web
by rezonant on Friday March 26, 2021 @10:24PM
(#26599304)
Will it be a flat fee, so that I, a lowly one-man crawler developer will not be able to afford it? Will it be that only Google can afford it, thus making their monopoly position even stronger?
Is there a Wikipedia crawling "welfare" program if I'm not a trillion dollar mega company?
Re:Only Google is really allowed to crawl the web
by dannyw on Friday March 26, 2021 @11:08PM
(#26599565)
Actually, yes! The current API is free. The new Enterprise API is paid.
Re:Only Google is really allowed to crawl the web
by aabhay on Saturday March 27, 2021 @12:07PM
(#26603290)
Sure! Apply to become a crawler. And if you meet certain criteria and your crawlers don’t exceed a quota then have at it. The key is not to make it technically challenging, but to erect a legal barrier.
No, but they often ask for donations when you visit the site, which people won't see if they just see the in-line blurb from Wikipedia on the Google results page.
> In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2].
$2M is a pittance compared to what I expect Google believes is the value of their Wikipedia blurbs. If Wikipedia could charge for use of this data (which another commenter claims they are working on doing), they could easily make orders of magnitude more money from Google.
Of course, my expectation is that Google would rather drop the Wikipedia blurbs entirely, or source the data elsewhere, than pay significantly more.