Re:Anyone got a contact at OpenAI. They have a spider problem
by Dylan16807 on Thursday April 11, 2024 @04:48PM
(#40006662)
How important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?
[ Parent
| on HN ]
Re:Anyone got a contact at OpenAI. They have a spider problem
by michaelt on Thursday April 11, 2024 @05:13PM
(#40006875)
Maybe not full full text search, but you'll generally want to remove the duplicates and suchlike.
Re:Anyone got a contact at OpenAI. They have a spider problem
by Dylan16807 on Thursday April 11, 2024 @05:28PM
(#40007012)
I guess you want some fast extra storage for as long as it takes to run https://github.com/chatnoir-eu/chatnoir-copycat but that's a very temporary thing.
Re:Anyone got a contact at OpenAI. They have a spider problem
by bevekspldnw on Thursday April 11, 2024 @08:47PM
(#40008339)
my guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.