News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by dragonwriter on Friday March 26, 2021 @12:25PM (#26593957)
> Perhaps there could be some kind of 'Crawler consortium'?

An industry-wide agreement not to compete for commercially valuable access to suppliers of data?

Comprised of companies that are current (and in some cases perennial) focusses of antitrust attention?

I think there might be a problem with that plan.

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by neolog on Friday March 26, 2021 @02:40PM (#26595407)
I don't see the problem. If a bunch of non-google companies pooled resources to make a crawl, that would reduce market concentration, not increase it.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by gnramires on Friday March 26, 2021 @12:34PM (#26594063)
Well, yes, and a common solution to anti-trust cases, that I know of, is some kind of industry self-regulation. In this case I wouldn't trust the industry only to self-regulate; hence, they should at invite (while keeping a minority but not insignificant position) governments and civil society (ngos and other organizations) to participate.

Could you better describe your objections?

[ Parent | on HN ]