News for hackers. Stuff that matters.
Re:Show HN: An open-source AI Gateway with integrated guardr... by roh26it on Wednesday August 14, 2024 @11:04AM (#41246949)
We use a bunch of caching mechanisms on the LLM requests themselves and extend the same to guardrails now.

So there's 2 levels of cache - the LLM request itself might be cached (simple and semantic) and the guardrail response can be cached as well.

We use a mix of a distributed kv store and a vector DB to actually store the data

There are no comments on this comment yet.