Re:FANN: Vector Search in 200 Lines of Rust
by phillipcarter on Thursday June 15, 2023 @01:48PM
(#36344129)
Our top_k search in our own product is in Go and it's ~25 lines of code. Super simple stuff, does the job of funding the top_k in 1ms, no index, no framework, nothing. We benefit from the corpus we embed being relatively static and we can just store them in redis, so it's not a hard problem to solve. Anyways, often you don't need some turnkey solution to this stuff.
[ Parent
| on HN ]
Re:FANN: Vector Search in 200 Lines of Rust
by nikhilgarg28 on Thursday June 15, 2023 @02:50PM
(#36344937)
What is the size and dimensionality of your corpus?
Re:FANN: Vector Search in 200 Lines of Rust
by phillipcarter on Thursday June 15, 2023 @04:37PM
(#36346179)
Size can be up to ~140MB of vectors, so not exactly enormous. But it's nowhere near enough to stress a single node running a for loop on the data and doing a dot product, then sorting by relevancy.
Re:FANN: Vector Search in 200 Lines of Rust
by zainhoda on Thursday June 15, 2023 @04:35PM
(#36346152)
Same — I just asked ChatGPT to generate Go code to do cosine similarity and I was surprised when the code it generated worked and the worst performance I get is like 10ms
Re:FANN: Vector Search in 200 Lines of Rust
by phillipcarter on Thursday June 15, 2023 @04:38PM
(#36346196)
Yep. If you parallelize with goroutines it'll be faster, but we're talking about shaving a few ms. The effectiveness of a for loop crunching floats in a single thread is not to be underestimated.
I have gotten 10x speedups with SIMD on modern hardware - goes from fast to blazing.
https://github.com/stillmatic/gollum/blob/main/vectorstore.g...
FWIW - Weaviate's pure Go implementation of cosine similarity is probably as fast as you can possibly get in pure go, without SIMD -- https://github.com/weaviate/weaviate/blob/2f44dbee52196656bd...
Re:FANN: Vector Search in 200 Lines of Rust
by phillipcarter on Thursday June 15, 2023 @06:02PM
(#36347258)
Yep, we'd go with what they did, except OpenAI embedding vectors are already normalized, so you can just do the dot product.
Re:FANN: Vector Search in 200 Lines of Rust
by nearestn on Thursday June 15, 2023 @08:39PM
(#36349259)
Weaviate actually does use SIMD for AMD64 with Go assembly (cosine distance is just normalized dot product) https://github.com/weaviate/weaviate/blob/master/adapters/re...
Re:FANN: Vector Search in 200 Lines of Rust
by AA-BA-94-2A-56 on Thursday June 15, 2023 @08:50PM
(#36349364)
Do you need anything special to run a Rust program with SIMD enabled?