News for hackers. Stuff that matters.
Re:Show HN: A lightweight AI gateway to 100+ models, in TS by retrovrv on Monday January 08, 2024 @04:08PM (#38918149)
Thank you! We have built out the cache system -- we do both simple caching (matching the request strings 100%) and also do semantic caching (returning a cache hit for semantically similar requests). More here - https://portkey.ai/docs/product/ai-gateway-streamline-llm-in...

The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!

Re:Show HN: A lightweight AI gateway to 100+ models, in TS by shaial on Wednesday January 10, 2024 @04:43AM (#38938285)
Awesome! We built the simple version in-house, and hoped someone would productize it.