News for hackers. Stuff that matters.

A comment on Anyone got a contact at OpenAI. They have a spider problem. Parent comment

Re:Anyone got a contact at OpenAI. They have a spider problem by michaelt on Thursday April 11, 2024 @05:13PM (#40006875)
Maybe not full full text search, but you'll generally want to remove the duplicates and suchlike.
[ Parent | on HN ]
Re:Anyone got a contact at OpenAI. They have a spider problem by Dylan16807 on Thursday April 11, 2024 @05:28PM (#40007012)
I guess you want some fast extra storage for as long as it takes to run https://github.com/chatnoir-eu/chatnoir-copycat but that's a very temporary thing.
[ Parent | on HN ]
Re:Anyone got a contact at OpenAI. They have a spider problem by bevekspldnw on Thursday April 11, 2024 @08:47PM (#40008339)
my guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.
[ Parent | on HN ]