News for hackers. Stuff that matters.
Re:VectorDB: Vector Database Built by Kagi Search by andy99 on Sunday November 26, 2023 @06:47AM (#38420910)
The two most interesting things to me in a "minimal" framework would be eliminating dependency on HF transformers and helping customize chunking.

Not a knock against this project, I see where it can be helpful.

Re:VectorDB: Vector Database Built by Kagi Search by dmezzetti on Sunday November 26, 2023 @06:52AM (#38420935)
My assumption is that this gives you the ability to locally encode vectors. This is useful for those not using API services to build their vectors.
Re:VectorDB: Vector Database Built by Kagi Search by andy99 on Sunday November 26, 2023 @07:25AM (#38421096)
Transformer inference is ~60 lines of numpy[0] (closer to 500 once you add tokenization etc). It would be nice to just have this and not all of pytorch and transformers.

[0] https://jaykmody.com/blog/gpt-from-scratch/

Re:VectorDB: Vector Database Built by Kagi Search by dmezzetti on Sunday November 26, 2023 @07:31AM (#38421126)
What about models besides GPT? Most of the popular vector encoding models aren't using this architecture.

If you really didn't want PyTorch/Transformers, you could consider exporting your models to ONNX (https://github.com/microsoft/onnxruntime).

Re:VectorDB: Vector Database Built by Kagi Search by cherryteastain on Sunday November 26, 2023 @07:41AM (#38421175)
It's 60 lines for CPU only inference, which'll be slow. If you want GPU acceleration it'll be a lot more than 60 lines.
Re:VectorDB: Vector Database Built by Kagi Search by jsimian on Sunday November 26, 2023 @01:04PM (#38423548)
Yeah, I just tried this out (props to the devs, super easy to set up) and my main gripe is the chunking algorithms aren't great - could be alot more useful with a context option that gives surrounding results to search results. The sliding window chunking method always cuts off the start of sentences.
Re:VectorDB: Vector Database Built by Kagi Search by andy99 on Sunday November 26, 2023 @01:56PM (#38423963)
I've found it works better to chunk by some logical sections in the document, e.g header h2 h3 h4 etc or 1.1 1.1.1 ... plus to be able to ignore some stuff (header and footer) plus other customizations.

At least for use cases where there are clusters of many similarly formatted documents, it would be cool to have a way of easily customizing chunking.