News for hackers. Stuff that matters.
Re:VectorDB: Vector Database Built by Kagi Search by andy99 on Sunday November 26, 2023 @07:25AM (#38421096)
Transformer inference is ~60 lines of numpy[0] (closer to 500 once you add tokenization etc). It would be nice to just have this and not all of pytorch and transformers.

[0] https://jaykmody.com/blog/gpt-from-scratch/

Re:VectorDB: Vector Database Built by Kagi Search by dmezzetti on Sunday November 26, 2023 @07:31AM (#38421126)
What about models besides GPT? Most of the popular vector encoding models aren't using this architecture.

If you really didn't want PyTorch/Transformers, you could consider exporting your models to ONNX (https://github.com/microsoft/onnxruntime).

Re:VectorDB: Vector Database Built by Kagi Search by cherryteastain on Sunday November 26, 2023 @07:41AM (#38421175)
It's 60 lines for CPU only inference, which'll be slow. If you want GPU acceleration it'll be a lot more than 60 lines.