A good middle ground could be to allow the diff to land once the “AI quick check” passed, then keep the full test suite running in the background. If they run them side by side for a while and see that the AI quick check caught the failing test every time, I’d be convinced.
At Meta we do similar heuristics on which tests to run per PR. When the system gets it wrong, which is often, its painful and leads to merged code that broke unrun tests
Like anything it's about tradeoffs. Though if it were me i'd simply write a mechanism which deterministically decides which areas of code pertain to which tests and use that simple program to determine which tests run. The algorithm could be loosely as simple as things like code owners and git-blame, but relative to some big set of Code->Test that you can have Claude/etc build ahead of time. The difference being it's deterministic between PRs and can be audited by humans for anything obviously poor or w/e.
As much as LLMs are interesting to me i hate using them in places where i want consistency. CI seems terrible for them.. to me at least.
Which suggests there might be big room for improvement. If LLMs open up new ways to be more precise about which E2E to run, that is a significant benefit, because you’d manage risk but reduce the time and resources it takes.
The opening part of the post makes certain claims -- "catch bugs before they ship" and "keep our master branch always clean" -- that can only be true if the coverage is not reduced. But, as other replies point out, if the heuristic for test selection has false negatives, the coverage is effectively reduced, which can result in bugs.
That problem can be bad enough when the heuristic is deterministic. AI throws determinism out of the window, which only exacerbates the problem.