News for hackers. Stuff that matters.

retrovrv

Re:Anthropic Services Down on Wednesday September 10, 2025 @02:11PM (#45201578)
Your best bet is having an account on AWS Bedrock & Vertex AI so you're able to route your request to the same model (such as claude-sonnet-4) but on a different provider.
there's portkey that we've been working on: https://github.com/portkey-AI/gateway
we essentially built the gateway as a service rather than an SDK: https://github.com/portkey-AI/gateway
Re:Is Outbound Going to Die? on Tuesday April 29, 2025 @03:28AM (#43829605)
just wanted to say, fantastic note. thanks for sharing!
thank you! we don't have a strong Evals module within our Prompt Studio at the moment. So there's no straightforward way to do that. however, we do have one of the better modules for applying guardrails on live AI traffic and setting up routing based on guardrail verdicts: https://portkey.ai/features/guardrails
lol. i get you - i think the 3rd point you shared - it's not exactly about CI issue - CI is already as fast as it can be. It's just that a prompt is a critical part of your AI app and any change to it needs to go through a few hoops before it's available to all users.

On Portkey, since we decouple prompt templates from your code - you can continue iterating on the prompts on Portkey and just reference the prompt ID in code. Any change you make to Portkey prompts automatically get reflected in the app because the prompt ID keeps pointing to the latest / published version.

does that make sense?

Show HN: Prompt Engineering Studio – Toolkit for deploying AI prompts at scale

Hey HN! Long-time reader, occasional commenter here. I wanted to share something my team and I have been building to solve our own frustrations.We've all seen the explosion of prompt engineering tools lately. While playing around in playgrounds is fun, when we tried to take our AI prompts to production, we hit a wall. I'm guessing many of you have experienced similar pain points.We kept hitting...

( Read More... | 4 comments | on HN )
Re:Garak, LLM Vulnerability Scanner on Monday November 18, 2024 @01:28AM (#42170235)
This is actually pretty cool. Would love to try it!
I'm affiliated with Portkey, so can answer who would need such a proxy/gateway:

Sidenote: Arch is def interesting!

A typical user we've seen at Portkey is a mid or a large size eng org where a central "Gen AI team" has now come up. This Gen AI team builds services that the rest of the company uses to build whatever AI features or products they want.

To build such a service, they need traditional API Gateway features like rate limiting, access rules, and also AI-specific features like universal API to multiple LLM providers, universal routing, central guardrails, AI-native observability + central dashboard for other stakeholders, and more.

It can absolutely be a plugin on top of existing Gateways.. like we've explored putting Portkey on Kong, but the need for a dedicated AI Gateway still remains, that can do all of these things I described in an easier way.

Probably, solutions like Langchain/Llamaindex etc. also fit in somewhere here, but a dedicated service for "ops" related issues for LLM APIs is something that we're seeing orgs adopt as a good practice.

came across this guide earlier - valuable insights. thanks for sharing!
there's an open source ai gateway - https://github.com/Portkey-AI/gateway
Phenomenal to see how Ragas has progressed. Congratulations on the launch
Re:Open-source tools for LLM monitoring and observability on Monday February 05, 2024 @05:43AM (#39259642)
https://portkey.ai/ - is also a nice addition to the list
Re:Show HN: A lightweight AI gateway to 100+ models, in TS on Monday January 08, 2024 @04:08PM (#38918149)
Thank you! We have built out the cache system -- we do both simple caching (matching the request strings 100%) and also do semantic caching (returning a cache hit for semantically similar requests). More here - https://portkey.ai/docs/product/ai-gateway-streamline-llm-in...

The caching part isn't open source yet, but part of our internal workers. Would be very cool to open source it!

Re:Show HN: A lightweight AI gateway to 100+ models, in TS on Monday January 08, 2024 @08:40AM (#38911742)
Pretty excited to announce this! While there are some popular and awesome AI gateways out there, like litellm, bricksai - none are written in TS, and for the TS ecosystem. Looking forward to the community's feedback
Re:OpenAI DevDay's Implications for LLM Apps in Prod on Thursday November 09, 2023 @01:46AM (#38201911)
I really liked request_format param to enforce JSON outputs, and seed param to enforce deterministic outputs - both I've already started to use.
Re:GPT-4 is Getting Faster on Wednesday October 18, 2023 @02:03PM (#37932367)
Thanks for sharing my blog here!

Quick notes on the analysis: - This is based on data from 100+ organizations globally, doing million+ requests a day via Portkey.ai. - I've randomly sampled 10,000 requests for both GPT 3.5 & 4 each day, for the past 3 months. - Variants like -0613 & -0314 are grouped under GPT3.5 & GPT4 for clarity. - I've plotted 'Latency PER Token' rather than just latency - For each day, I've counted percentile values for Latency/Token across different percentiles like 50, 75, 90, 99 to account for variance in prompt length and complexities and separate out anomalies

Happy to share more or answer questions, if any.

Re:Show HN: Magentic – Use LLMs as simple Python functions on Tuesday September 26, 2023 @02:37PM (#37663858)
Super cool! Looks quite intuitive, especially for function calls.
There are quite a few LLM monitoring tools in the market. But for monitoring (or evaluating) RAG systems, I found Ragas to be the most helpful: https://blog.langchain.dev/evaluating-rag-pipelines-with-rag...
Re:Show HN: AI Grants and Credits Finder on Wednesday August 16, 2023 @02:12PM (#37151242)
There are some programs that match founders, bring in VCs and overall facilitate "Networking" - those are tagged as such. Hope that helps!
Re:Ask HN: Where to Host Llama 2? on Monday August 14, 2023 @05:41PM (#37127167)
Looks very interesting
Re:Ask HN: Is there a market for premium Figma plugins? on Monday August 14, 2023 @12:27AM (#37117458)
These guys built a few Figma plugins, made money from that, and also got acquired by Figma: https://diagram.com/
Re:Ask HN: What ChatGPT plugins do you use daily, if any? on Monday August 14, 2023 @12:25AM (#37117445)
Code interpreter, though not exactly a plugin, has become my default mode of interacting with ChatGPT.
A lot of thought has been put into this. Quite like the dashboard design, and the simplicity of using this. Congrats on the launch!
Re:Show HN: Q&A with AI Trained on Bankruptcy Law on Friday August 11, 2023 @04:25AM (#37086356)
My first question as well, especially considering how fast it is. But understand if you can't reveal.

Would still be cool to learn technical details.

Re:Ask HN: Where do people sell apps these day? on Friday August 11, 2023 @04:12AM (#37086294)
To add to what others have pointed out, if you have a good following on any social platform and can push your product there, putting it behind something like Gumroad is also an option.

Older »