News for hackers. Stuff that matters.
Re:Common Crawl by riedel on Friday March 26, 2021 @03:16PM (#26595810)
As a European (German) I am always wondering about the legal basis for a) making a copy of copyrighted material and databases available and b) processing contained personal data.

The Internet archive seems more like a library with exceptions applying, but common crawl seems to advertise also many other purposes that go beyond archiving publically relevant content.

Would this be possible in Europe, too? My feeling is that US legislation different here. Do you have to actively claim copyright in the US or enforce technically e.g. via DRM? Anyone can use anything without a license as long nobody finds out?

Re:Common Crawl by sennight on Friday March 26, 2021 @08:02PM (#26598455)
Well they've got a bunch of different data that would fall under different precedents. Metadata would likely fall under the phone book ruling. The raw warc data could be argued to be transformative, and would certainly fall under fair use - especially because it isn't even close to threatening market replacement. Property abandonment might fit in somewhere as well :)

https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,....

Re:Common Crawl by nodamage on Friday March 26, 2021 @07:22PM (#26598187)
I am curious about this too, from a copyright perspective:

1) Whether it is legal for Common Crawl to collect this data in the first place? Presumably they don't have explicit permission from every site they have crawled?

and

2) Whether it is legal to use this data to train ML algorithms for commercial use (e.g. GPT-3)? They provide a bunch of examples of this on their site but the terms of use are kind of vague about this.

Re:Common Crawl by nonameiguess on Saturday March 27, 2021 @09:37AM (#26602205)
It has to be legal to use legally acquired published information for a commercial purpose. That is not nearly the same thing as selling someone else's work. Training a ML algorithm on published work is no different from me reading a tutorial on the web and using the new skills I learned to do something that makes me money. That is very different from reselling the tutorial as if I authored it.
Re:Common Crawl by visarga on Friday March 26, 2021 @11:35PM (#26599699)
The data is open they don't crawl inside password protected areas of the internet. Why shouldn't they be able to read data put into the open? If the authors didn't want their data openly accessible they should put it in a private place.
Re:Common Crawl by nodamage on Saturday March 27, 2021 @01:31AM (#26600165)
Except just because someone published something on the internet doesn't mean you have the legal right to reproduce and redistribute it yourself.
Re:Common Crawl by fjeifisjf on Saturday March 27, 2021 @08:47AM (#26601921)
That has nothing to do with reading data that the publisher does distribute.
Re:Common Crawl by nodamage on Sunday March 28, 2021 @06:53PM (#26615567)
They're not just reading data, they are redistributing it.