the full-session evaluation framing resonates a lot. we've been running browser automation agents and the exact same problem shows up: individual actions pass, session fails. a click works, a form fill works, but the session gets flagged or blocked 8 turns in because something earlier created a signal that compounded.

one failure mode that's specific to browser agents and doesn't get much attention: the test environment is too clean. when you run simulations against a controlled setup, the agent never encounters the friction that real sessions do - bot detection challenges, CAPTCHAs, dynamic content that loads differently, fingerprinting checks mid-session. so you end up with agents that pass your test suite but fail in the wild, and the gap is in the environmental assumptions not the agent logic.

the mock tool platform approach is interesting precisely because it sidesteps this - you're testing the agent's decision-making in isolation from the messy runtime. that's valid for catching logic regressions. but i'd be curious how you handle cases where the tool call itself triggers secondary effects in the environment (e.g. the API call changes session state in ways that affect what the agent sees next).

also, does your session-level judge handle cases where the correct behavior is adaptive - where the agent should change strategy mid-session based on what it encountered? that feels like a harder eval problem than a fixed expected outcome.