News for hackers. Stuff that matters.

A comment on The Pile is a 825 GiB diverse, open source language modelling data set. Parent comment

Re:The Pile is a 825 GiB diverse, open source language model... by seanhunter on Thursday March 07, 2024 @01:49PM (#39633047)
Indeed. Also in the US, whether or not something is fair use involves a four factor test[1] and two of the factors are the amount and substantiality of what's taken and the effect on any market. In this case, the amount is "everything" and the effect on the market is potentially very large for authors/publishers.

[1] https://fairuse.stanford.edu/overview/fair-use/four-factors/

[ Parent | on HN ]
Re:The Pile is a 825 GiB diverse, open source language model... by fsckboy on Thursday March 07, 2024 @02:05PM (#39633239)
>two of the factors are the amount and substantiality of what's taken and the effect on any market

books.google.com has been allowed to copy all the books they can lay their hands on, so long as they don't regurgitate them in full, so it's not really the taking, but any subsequent reproductions. And the effect on the market is insubstantial if the alternative wasn't going to be the equivalent sales.

[ Parent | on HN ]
Re:The Pile is a 825 GiB diverse, open source language model... by ascorbic on Thursday March 07, 2024 @05:06PM (#39635134)
You can download the whole dataset, so they're certainly able to regurgitate them in full.
[ Parent | on HN ]