News for hackers. Stuff that matters.

A comment on Vector indexing all of Wikipedia on a laptop. Parent comment

Re:Vector indexing all of Wikipedia on a laptop by thomasfromcdnjs on Wednesday May 29, 2024 @05:36PM (#40517307)
Thanks for the answer.

Wikipedia has a lot of tables so I was wondering if content-aware sentence chunking would be good enough for Wikipedia.

https://www.pinecone.io/learn/chunking-strategies/

[ Parent | on HN ]
Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @07:22PM (#40518303)
mwparserfromhell can parse the text content without including tables
[ Parent | on HN ]