News for hackers. Stuff that matters.

A comment on Vector indexing all of Wikipedia on a laptop. Parent comment

Re:Vector indexing all of Wikipedia on a laptop by rahimnathwani on Wednesday May 29, 2024 @04:24PM (#40516587)
Why would you want chunks that big for vector search? Wouldn't there be too much information in each chunk, making it harder to match a query to a concept within the chunk?
[ Parent | on HN ]
Re:Vector indexing all of Wikipedia on a laptop by nostrebored on Wednesday May 29, 2024 @11:57PM (#40519928)
The problem is that often semantic meaning depends on state multiple paragraphs or sections away.

This is a coarse way to tackle that

[ Parent | on HN ]