retrovrv
retrovrv on Hacker News
Your best bet is having an account on AWS Bedrock & Vertex AI so you're able to route your request to the same model (such as claude-sonnet-4) but on a different provider.
just wanted to say, fantastic note. thanks for sharing!
thank you! we don't have a strong Evals module within our Prompt Studio at the moment. So there's no straightforward way to do that. however, we do have one of the better modules for applying guardrails on live AI traffic and setting up routing based on guardrail verdicts:
https://portkey.ai/features/guardrails
lol. i get you - i think the 3rd point you shared - it's not exactly about CI issue - CI is already as fast as it can be. It's just that a prompt is a critical part of your AI app and any change to it needs to go through a few hoops before it's available to all users.
On Portkey, since we decouple prompt templates from your code - you can continue iterating on the prompts on Portkey and just reference the prompt ID in code. Any change you make to Portkey prompts automatically get reflected in the app because the prompt ID keeps pointing to the latest / published version.
does that make sense?
Posted by
retrovrv on Tuesday March 18, 2025 @03:52AM
from the show-hn dept.
Hey HN! Long-time reader, occasional commenter here. I wanted to share something my team and I have been building to solve our own frustrations.We've all seen the explosion of prompt engineering tools lately. While playing around in playgrounds is fun, when we tried to take our AI prompts to production, we hit a wall. I'm guessing many of you have experienced similar pain points.We kept hitting...
This is actually pretty cool. Would love to try it!
I'm affiliated with Portkey, so can answer who would need such a proxy/gateway:
Sidenote: Arch is def interesting!
A typical user we've seen at Portkey is a mid or a large size eng org where a central "Gen AI team" has now come up. This Gen AI team builds services that the rest of the company uses to build whatever AI features or products they want.
To build such a service, they need traditional API Gateway features like rate limiting, access rules, and also AI-specific features like universal API to multiple LLM providers, universal routing, central guardrails, AI-native observability + central dashboard for other stakeholders, and more.
It can absolutely be a plugin on top of existing Gateways.. like we've explored putting Portkey on Kong, but the need for a dedicated AI Gateway still remains, that can do all of these things I described in an easier way.
Probably, solutions like Langchain/Llamaindex etc. also fit in somewhere here, but a dedicated service for "ops" related issues for LLM APIs is something that we're seeing orgs adopt as a good practice.
came across this guide earlier - valuable insights. thanks for sharing!
Phenomenal to see how Ragas has progressed. Congratulations on the launch
Thank you! We have built out the cache system -- we do both simple caching (matching the request strings 100%) and also do semantic caching (returning a cache hit for semantically similar requests). More here -
https://portkey.ai/docs/product/ai-gateway-streamline-llm-in...The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!
Pretty excited to announce this! While there are some popular and awesome AI gateways out there, like litellm, bricksai - none are written in TS, and for the TS ecosystem. Looking forward to the community's feedback
Posted by
retrovrv on Wednesday January 03, 2024 @07:12AM
from the portkey.ai dept.
I really liked request_format param to enforce JSON outputs, and seed param to enforce deterministic outputs - both I've already started to use.
Thanks for sharing my blog here!
Quick notes on the analysis:
- This is based on data from 100+ organizations globally, doing million+ requests a day via Portkey.ai.
- I've randomly sampled 10,000 requests for both GPT 3.5 & 4 each day, for the past 3 months.
- Variants like -0613 & -0314 are grouped under GPT3.5 & GPT4 for clarity.
- I've plotted 'Latency PER Token' rather than just latency
- For each day, I've counted percentile values for Latency/Token across different percentiles like 50, 75, 90, 99 to account for variance in prompt length and complexities and separate out anomalies
Happy to share more or answer questions, if any.
Posted by
retrovrv on Monday October 16, 2023 @11:57PM
from the portkey.ai dept.
Posted by
retrovrv on Tuesday September 26, 2023 @02:57PM
from the github.com dept.
Super cool! Looks quite intuitive, especially for function calls.
There are some programs that match founders, bring in VCs and overall facilitate "Networking" - those are tagged as such. Hope that helps!
These guys built a few Figma plugins, made money from that, and also got acquired by Figma:
https://diagram.com/
Code interpreter, though not exactly a plugin, has become my default mode of interacting with ChatGPT.
A lot of thought has been put into this. Quite like the dashboard design, and the simplicity of using this. Congrats on the launch!
Posted by
retrovrv on Sunday August 13, 2023 @11:58PM
from the ask-hn dept.
There's ollama/ggml etc for local setup, but other than Replicate, what are the other options for hosting Llama 2?
My first question as well, especially considering how fast it is. But understand if you can't reveal.
Would still be cool to learn technical details.
To add to what others have pointed out, if you have a good following on any social platform and can push your product there, putting it behind something like Gumroad is also an option.