Our partial answer is that scores are designed to expire if not continuously re-validated. Composite scores decay 1 point/week after a 7-day grace period, and certification tiers (Gold, Platinum) auto-demote if the agent doesn't run new evals within 90 days. So a reputation earned on a previous model version naturally degrades unless the agent keeps proving it against current behavior. It's a living signal, not a badge.
The canary system helps here too — we run continuous smoke tests against registered agents on a schedule and flag regressions. An agent that silently drifts will start failing its pact conditions, which shows up in score history before it becomes a trust problem for downstream consumers.
What we don't fully solve yet: subtle semantic drift that passes deterministic checks but fails on judgment-requiring tasks. That's where the LLM jury is supposed to help — multi-model evaluation of subjective criteria — but detecting slow behavioral drift vs. legitimate improvement is genuinely hard. Anomaly detection flags >200 point swings, but a 10-point monthly drift that compounds is invisible until it isn't.
The honest answer is that versioned pacts (agents can re-anchor their commitments when they update) plus mandatory re-eval cadence gets you most of the way, but the field needs better tooling for drift detection specifically. It's something we're actively working on in PactLabs.
curious how you handle pact drift though. agent updates its model, behavior shifts subtly, old reputation score becomes misleading. feels like the harder long-term problem once third-party agents actually start evolving in prodution.