vishaal_007

We kept running into the same wall. The prototype worked great: chain a few calls together, get a response and it felt like we were basically done. Then real traffic showed up and we were spending more time on the stuff around the LLM call than the call itself.

Retries, failover when a provider went down, figuring out why latency quietly doubled over a week, parsing responses that were slightly different between models, paying full token cost for the same request hitting us over and over. Each one felt like a small fix. A few months later it was a pile of glue code nobody wanted to touch.

At one point we had a Slack bot that pinged us when latency crossed 5 seconds. It mostly taught us how often latency crosses 5 seconds.

This page is basically about where that led us. Not saying LangChain is the wrong tool, if you want maximum flexibility, it still makes a lot of sense. This is more for teams that got something working quickly and then realised prod was a completely different problem.

Curious to hear from anyone else who’s gone through the same thing. What did you end up building or buying to deal with it?

I’m one of the people behind ModelRiver.

We wrote this after running into the same pattern a few times: the AI feature worked fine in development, but once real traffic hit it, the problems were mostly infra problems, not prompt problems. Provider outages, repeated token spend on identical requests, poor visibility into failures, and response shape drift.

This post is our attempt to explain the architecture pattern we ended up with, before talking about the product itself. The short version is: put a routing layer between your app and the model providers, keep the client interface the same, and move failover/cache/observability/response validation into that layer.

If you’ve built around direct OpenAI calls and had to patch retries/fallbacks/logging yourself, I’d be interested to hear what broke first for you.

Co-founder here—it's just me and my partner bootstrapping this thing. We've been wasting tokens left and right just trying to debug our response parsing code. Not even the AI logic, mind you, just our own sloppy stuff and don't get me started on CI: tests flaking out because GPT decides to rephrase something randomly. We got fed up paying real money to fix our bugs, so we hacked together a "Test Mode”. It routes your calls through the full pipeline (auth, logging, everything) but swaps in your sample data instead of hitting the actual provider. No tokens burned, totally deterministic, and lightning fast.

How are folks handling testing for AI integrations? Mocking always felt half-baked to us since it bypasses the real flow. What's working for you?

In our experience, it usually comes down to whether the request has user-visible state over time. If the response is something you can treat as atomic and either succeed or fail cleanly, it tends to stay simple.

The requests that “grow” tend to share a few signals early on: they stream partial results, they take long enough that the frontend needs progress updates, or failures start happening after something has already been shown to the user. Another common signal is when retries stop being transparent and you start needing to explain to users what actually happened.

Once those patterns show up, teams usually end up reworking the flow anyway. The event-driven approach just makes that lifecycle explicit earlier, instead of letting it emerge implicitly and painfully over time.

In practice, event-driven starts to feel like overkill when requests are short-lived and failures are cheap. If a call is fast, idempotent, and the user isn’t waiting on partial output, a simple sync request is usually the clearest solution.

Queue-based async still works well for batch jobs, offline processing, or anything where latency and ordering aren’t user-visible. The event-driven approach mainly pays off once you have long-lived or interactive requests where failures can happen mid-response and you care about what the user actually sees.

I’m another founder on this. One thing that surprised us while building AI features was how often the hard problems weren’t about model choice, but about request lifecycle. Once you introduce streaming, retries, and multiple providers, a lot of implicit assumptions in typical request–response code stop holding.

We kept seeing teams reinvent similar patterns in slightly different ways, especially around correlating events, handling partial failures, and keeping the frontend in sync with what actually happened on the backend. The goal with this writeup was to make those tradeoffs explicit and show what’s actually happening on the wire in each approach.

Curious to hear how others here are handling long-lived or streaming AI requests in production, especially once things start failing in non-obvious ways.