Re:You just do not need a specialized vector database
by mergejoin on Saturday August 12, 2023 @06:01PM
(#37104629)
i was _very_ surprised by that number as well. 10s for inference on 300 chars? using an entire A100 GB GPU? that does not make any sense. OpenAI and other companies would NEVER be able to scale to 100s of millions of users if that was the real number, right?
Is that right? That sounds like, 2 OOMs higher than I would've expected. Is he doing something wrong like loading the model from a cold start?