Anyone got a contact at OpenAI. They have a spider problem

Re:Anyone got a contact at OpenAI. They have a spider problem by bevekspldnw on Thursday April 11, 2024 @02:37PM (#40005338)

Gonna say you’re way off there. Once you decompress common crawl and index it for FTS and put it on fast storage you’re in for some serious pain, and that’s before you even put it in your ML pipeline.

Even refined web runs about 2TB once loaded into Postgres with TS vector columns, and that’s a substantially smaller dataset than common crawl.

It’s not just a dumping a to of zip files on your NAS, it’s making the data responsive and usable.

[ Parent | on HN ]

Re:Anyone got a contact at OpenAI. They have a spider problem by Dylan16807 on Thursday April 11, 2024 @04:48PM (#40006662)

How important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?

[ Parent | on HN ]

Re:Anyone got a contact at OpenAI. They have a spider problem by michaelt on Thursday April 11, 2024 @05:13PM (#40006875)

Maybe not full full text search, but you'll generally want to remove the duplicates and suchlike.

[ Parent | on HN ]

Re:Anyone got a contact at OpenAI. They have a spider problem by Dylan16807 on Thursday April 11, 2024 @05:28PM (#40007012)

I guess you want some fast extra storage for as long as it takes to run https://github.com/chatnoir-eu/chatnoir-copycat but that's a very temporary thing.

[ Parent | on HN ]

Re:Anyone got a contact at OpenAI. They have a spider problem by bevekspldnw on Thursday April 11, 2024 @08:47PM (#40008339)

my guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.

[ Parent | on HN ]