Re:FineWeb: Decanting the web for the finest text data at scale
by nmstoker on Sunday June 02, 2024 @04:57AM
(#40552591)
Am continuing to reflect on this, so may have more to say later, but what struck me first was the commitment to openness with FineWeb, it is impressive and thorough (eg all the scripts they actually used are available and linked to, not just the finished data)
[ Parent
| on HN ]
Re:FineWeb: Decanting the web for the finest text data at scale
by sieszpak on Sunday June 02, 2024 @09:17AM
(#40553867)
I agree, the information provided in this article is a treasure. Maybe someone will add some "magic sauce" to it?
Re:FineWeb: Decanting the web for the finest text data at scale
by nmstoker on Sunday June 02, 2024 @11:13AM
(#40554821)
The deduplication discussion shows they don't filter out ads as part of their cleaning - I appreciate this could be risky and perhaps a huge processing step given dataset sizes, but intuitively it feels like it would cut the noise dramatically and thus help tbe signal within datasets.