News for hackers. Stuff that matters.
Re:Rag is more than just embedding search by binarymax on Thursday September 21, 2023 @02:08PM (#37601561)
:waves: Hi Doug! (he's co-author of Relevant Search and contributing author of AI Powered Search too)

That's definitely a thing. But alarms go off in my head when I think about query latency and cost. Can't imagine running 1k qps while sending every single one to GPT or LLama - thats the stuff of production nightmares for me!

If you've got less demand and have a couple queries a second, then maybe it's OK - but you're probably adding a good second on top of your query latency.

There are no comments on this comment yet.