News for hackers. Stuff that matters.

A comment on The Pile is a 825 GiB diverse, open source language modelling data set. Parent comment

Re:The Pile is a 825 GiB diverse, open source language model... by SEGyges on Thursday March 07, 2024 @03:52PM (#39634452)
You are uploading 5 billion examples of <something>. You cannot filter it manually, of course, because there are five billion of it. Given that it is the year 2024, how hard is it to be positive that a well-resourced team at Stanford in 2029 will not have better methods of identifying and filtering your data, or a better reference dataset to filter it against, than you do presently?

It is a pretty hard problem.

[ Parent | on HN ]
Re:The Pile is a 825 GiB diverse, open source language model... by vineyardmike on Thursday March 07, 2024 @05:58PM (#39635679)
You don’t have to do it manually. There is a database of file hashes.

And this isn’t just “one engineer”. Companies like StabilityAI, Google, etc have used LAION datasets. If you built a dataset you should expend some resources on automated filtering. Don’t include explicit imagery as an intentional choice if you can’t do basic filtering.

[ Parent | on HN ]