Vector indexing all of Wikipedia on a laptop

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @02:00PM (#40514827)

Nothing too crazy, just downloading a dump, splitting it into manageable batch sizes, and using a lightweight embedding model to vectorize each article. Using the best GPU available on colab it takes maybe 8 hours if I remember correctly? Vectors can be saved as NPY files and loaded into something like FAISS for fast querying.

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by abetusk on Wednesday May 29, 2024 @02:54PM (#40515457)

This probably deserves its own article and might be of interest to the HN community.

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @05:12PM (#40517080)

I will probably make a post when I launch my app! For now I’m trying to figure out how I can host the whole system for cheap because I don’t anticipate generating much revenue

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by solarengineer on Thursday May 30, 2024 @01:18AM (#40520320)

I'm interested in hearing about what you will be hosting.

Would Digital Ocean or Hetzner meet your needs?

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Thursday May 30, 2024 @02:31PM (#40527140)

I was going to use ec2 and s3, should I look at digital ocean or hetzner instead?

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by riku_iki on Wednesday May 29, 2024 @04:02PM (#40516336)

What is the end task(e.g. RAG, or just vector search for question answering), are you satisfied with results in terms of quality?

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @05:14PM (#40517094)

The end result is a recommendation algorithm, so basically just vector similarity search (with a bunch of other logic too ofc). The quality is great, and if anything a little bit of underfitting is desirable to avoid the “we see you bought a toilet seat, here’s 50 other toilet seats you might like” effect.

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by j0hnyl on Wednesday May 29, 2024 @02:17PM (#40515022)

How big is the resulting vector data?

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @02:40PM (#40515272)

Like 8 gb roughly

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by thomasfromcdnjs on Wednesday May 29, 2024 @04:54PM (#40516918)

Did you chunk the articles? If so, in what way?

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @05:17PM (#40517134)

Yes. I split the text into sentence and append sentences to a chunk until the max context window is reached. The context window size is dynamic for each article so that each chunk is roughly the same size. Then I just do a mean pool of the chunks for each article.

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by thomasfromcdnjs on Wednesday May 29, 2024 @05:36PM (#40517307)

Thanks for the answer.

Wikipedia has a lot of tables so I was wondering if content-aware sentence chunking would be good enough for Wikipedia.

https://www.pinecone.io/learn/chunking-strategies/

[ Parent | on HN ]

Re:Vector indexing all of Wikipedia on a laptop by gfourfour on Wednesday May 29, 2024 @07:22PM (#40518303)

mwparserfromhell can parse the text content without including tables

[ Parent | on HN ]