News for hackers. Stuff that matters.
Re:If you don't run your own LLM proxy, how do you even know... by CharlieDigital on Monday October 19, 2026 @07:38AM (#50151130)
You don't need a proxy to do this.

Both Codex and Claude Code output OTEL traces that you can ingest and process.

Other harnesses like OpenCode, Pi, etc. you can hook into their plugin architecture and implement your own OTEL emission from a plugin (or use an off the shelf plugin[0]).

Host an OpenTelemetry collector[1][2] or rely on your existing collector infra.

The OTEL traces will have exactly what you need. Tool calls, full prompts, token usage/cost, etc. It is less intrusive and does not become a single-point-of failure since the OTEL emission and collection is best-effort whereas a failing/overloaded proxy is going to strand your team until resolved.

--

[0] https://github.com/DEVtheOPS/opencode-plugin-otel

[1] Open source collector: https://opentelemetry.io/docs/collector/

[2] Sample configuration for filtering noise: https://github.com/zeeq-ai/zeeq-app/blob/main/build/otel-col...

Re:If you don't run your own LLM proxy, how do you even know... by InsidiousWombat on Monday October 19, 2026 @07:48AM (#50151222)
That shows what you do, not what the proxy does with your queries
Re:If you don't run your own LLM proxy, how do you even know... by CharlieDigital on Monday October 19, 2026 @08:00AM (#50151325)
If you are using a Datadog, Langfuse, Langchain, or other OTEL target, you do nothing. Just query it. An OTEL collector deployment is still useful for filtering.

If you want to deploy your own collector and write your own handler (e.g. put it into a database), you can generate the stubs for the OTEL payloads and convert to domain records.

Re:If you don't run your own LLM proxy, how do you even know... by T-RN-R on Monday October 19, 2026 @09:09AM (#50151997)
Disclosure: I work on Origin [1]. We built a platform for enterprises that consumes OTEL, and other traces without a proxy or an LLM gateway. We even support collecting traces from local inference workloads.

For supported tools, we collect prompts, responses, and tool calls, then tie them to the user, organizational context and what actually happened on the machine. We can then use this data to detect when agents (or the people piloting them) do bad things [2].

--

[1] https://www.originhq.com/

[2] https://www.originhq.com/investigations

Re:If you don't run your own LLM proxy, how do you even know... by ProjectBarks on Monday October 19, 2026 @10:41AM (#50153072)
Wouldn't a fair criticism of OTEL be that you only see what these harnesses are willing to show you?

There have been numerous scandals of AI companies offering CC/cursor alternatives, then indexing data they never should have; specifically Z.ai [0], ByteDance Trae [1] and Alibaba Qwen [2]. Maybe the argument should be that all harnesses should be sandboxed and monitored for all networking.

If it were me, all AI harnesses would be running on a VM with network logging enabled. Beyond not being able to trust what data they send via the harness, you should audit what data they access.

[0] https://www.chosun.com/english/industry-en/2026/09/22/ABYLC3...

[1] https://github.com/segmentationf4u1t/trae_telemetry_research

[2] https://github.com/QwenLM/qwen-code/issues/11198

Re:If you don't run your own LLM proxy, how do you even know... by JustFinishedBSG on Monday October 19, 2026 @12:42PM (#50154600)
Well if you own the harness ( and you should ) you see everything you want.

But I’ve seen approaches using eBPF to collect telemetry to OTEL

Re:If you don't run your own LLM proxy, how do you even know... by dinm on Monday October 19, 2026 @01:17PM (#50155121)
Sometimes, it gets to very hacky solutions, such as what we did at Laminar with Claude Agent SDK. Basically we spin up a tiny in-process proxy that intercepts requests and responses and emits the telemetry as OTEL traces. We repoint the Python / JS SDK to convince the harness that our proxy is the anthropic URL, and the proxy then forward the request downstream. I wrote about it some time ago [1]. This is definitely not elegant, but it generally works.

[1] https://laminar.sh/blog/2025-12-03-claude-agent-sdk-instrume...

Re:If you don't run your own LLM proxy, how do you even know... by pavelmelnichuk on Monday October 19, 2026 @12:05PM (#50154093)
Otel is only a quarter of what the llm proxies solve for you. What happens when an agent goes down some random rabbit hole and burns through $100 before you even notice? It's nice to have post-disaster visibility with Otel, but even self hosting an LLM proxy prevents the disaster.

Shameless plug, I've been building Relay for this exact reason. You get real-time visibility into what your agents are doing, how much they're spending, and can set hard limits so things don't spiral out of control. Plus a lot of other features they bring to the table if you want.

It's open source and self-hostable too.

https://github.com/anchorshell/relay https://anchorshell.com/

Re:If you don't run your own LLM proxy, how do you even know... by CharlieDigital on Monday October 19, 2026 @01:15PM (#50155069)
You can do that with MCP or hooks as well.
Re:If you don't run your own LLM proxy, how do you even know... by pavelmelnichuk on Monday October 19, 2026 @07:26PM (#50159367)
not really, for example relay gives you model "groups" ie: fallbacks if one is down, or prefer to use a free-endpoint while you have daily capacity, etc.

Maybe you need a decision model to route, or maybe you want to have guardrail-type models pre or post response for an enterprise client.

Re:If you don't run your own LLM proxy, how do you even know... by CharlieDigital on Tuesday October 20, 2026 @05:40AM (#50163530)
Re:If you don't run your own LLM proxy, how do you even know... by lokar on Monday October 19, 2026 @01:58PM (#50155646)
You can build a proxy setup that is not a SPOF.
Re:If you don't run your own LLM proxy, how do you even know... by CharlieDigital on Monday October 19, 2026 @02:26PM (#50156016)
Sure, but OTEL has all of the right qualities for the majority of use cases here.

Standards based, many off-the-shelf providers, out-of-process + buffered by default, many standard processors for traces + spans, many tools for storing and querying.

OTEL is pretty widely supported and mostly "just works" at this stage of maturity. A failing/mis-configured OTEL endpoint/collector isn't going to stop you from getting work done in the same way that a failing proxy is. You'll drop some traces, but that's probably OK in most cases.

There is a place for proxies, but those places should share the same failure boundary mode. Meaning if the proxy fails, it should necessarily stop forward agent activity. Example: proxy injection of secrets instead of local acquisition.

Re:If you don't run your own LLM proxy, how do you even know... by lokar on Monday October 19, 2026 @02:54PM (#50156389)
IMO, a key place for proxies is resource access where the client is not fully trusted (or for any reason it may fail to log correctly).

AI agents seem like a pretty clear case of this. Both because of recent issues, and that they tend to run on end user devices where strict compliance is harder to maintain.