The Pile is a 825 GiB diverse, open source language modelling data set

Re:The Pile is a 825 GiB diverse, open source language model... by pk-protect-ai on Thursday March 07, 2024 @02:35PM (#39633603)

I wish it had included the books3, but it doesn't anymore. I wish it was possible to download that 36GB books3.tar in the wild these days. Herewith, I promise to use this dataset according to the "fair use" only...

[ Parent | on HN ]

Re:The Pile is a 825 GiB diverse, open source language model... by SekstiNi on Thursday March 07, 2024 @03:29PM (#39634225)

> I wish it was possible to download that 36GB books3.tar in the wild these days.

There... is a torrent.

[ Parent | on HN ]

Re:The Pile is a 825 GiB diverse, open source language model... by pk-protect-ai on Thursday March 07, 2024 @04:32PM (#39634822)

I know. But here where I am, using torrent means participate in distribution of the content and that is where I'll get huge bill for illegally sharing this file.

[ Parent | on HN ]

Re:The Pile is a 825 GiB diverse, open source language model... by MacsHeadroom on Thursday March 07, 2024 @11:12PM (#39637688)

Use a debrid provider or seedbox to download the torrent. They torrent it for you and then you direct download from them. Should cost $10 or less.

[ Parent | on HN ]