We built Agent Compass after hitting the same wall over and over: agents generate thousands of traces with branching tool calls and no obvious pattern. APMs (Datadog/New Relic) tell you infra health. LLM observability tools (LangSmith/Arize) show trace detail. But the gap remained: turning all that data into a fast, defensible root cause and a concrete fix.
What it is
Automatic error clustering for AI agents
Symptom → likely root cause mapping
Actionable fix suggestions you can validate with a focused eval loop
Why it’s different
You debug categories of failures, not one-off traces
It ranks hypotheses (e.g., threshold too high, retrieval drift, prompt regression, guardrail friction)
It proposes small, surgical changes you can A/B and roll back quickly
How it works (high level)
We instrument LLM calls, tool invocations, retrieval hits, guardrail events, and outputs as spans (OpenTelemetry compatible)
We build semantic signatures of failure states and cluster them
We label clusters and map them to ranked hypotheses using a mix of rules + learned patterns from historical fixes
We attach a minimal eval set per cluster so you can confirm the fix without re-running your whole suite
What it is
Automatic error clustering for AI agents
Symptom → likely root cause mapping
Actionable fix suggestions you can validate with a focused eval loop
Why it’s different
You debug categories of failures, not one-off traces
It ranks hypotheses (e.g., threshold too high, retrieval drift, prompt regression, guardrail friction)
It proposes small, surgical changes you can A/B and roll back quickly
How it works (high level)
We instrument LLM calls, tool invocations, retrieval hits, guardrail events, and outputs as spans (OpenTelemetry compatible)
We build semantic signatures of failure states and cluster them
We label clusters and map them to ranked hypotheses using a mix of rules + learned patterns from historical fixes
We attach a minimal eval set per cluster so you can confirm the fix without re-running your whole suite