News for hackers. Stuff that matters.
Re:How the Claude Code team thinks about verification by FartyMcFarter on Tuesday November 17, 2026 @06:56AM (#50530251)
> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.

If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).

If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.

Re:How the Claude Code team thinks about verification by octoberfranklin on Tuesday November 17, 2026 @07:39PM (#50540338)
Nothing is ever perfectly hermetic.

Unless your code is 100% single-threaded you'll always have variable timing between threads (influenced by things like caches, etc).

Tests need to be not just hermetic but also deterministic. The problem is that determinism doesn't have a glib "just wrap it in a sandbox bro" way to achieve it.

Re:How the Claude Code team thinks about verification by FartyMcFarter on Wednesday November 18, 2026 @09:03AM (#50546461)
> Nothing is ever perfectly hermetic.

Strictly speaking that's true, since there's hardware involved.

But if you introduce reasonable assumptions such as having non-faulty hardware on the vast majority of test runs, most unit tests are in fact perfectly hermetic and deterministic for all practical purposes.

Unit tests are usually single-threaded too, which cuts off a huge source of nondeterminism. Integration tests can quickly get more messy though.