Re:GIMP now has UI tests
by timcambrant on Friday November 20, 2026 @12:41AM
(#50570588)
AI is a bit too excellent at writing tests. The number of tests doesn't really convey any information on test coverage. At least I've found that agents that naively improve codebases will write tests for pretty much anything, including irrelevant behavior that has little value in ensuring stability, security or correct behavior. It would probably help to run a test review stage with a strong model, where agents attempt to correct and consolidate tests to guard behavior rather than ensuring that theoretical code paths are followed.
[ Parent
| on HN ]
In my experience, designing tests cases to ensure that they're othogonal and verify your invariants (domain or technical) is very mentally taxing and not something you can truly delegate. The actual typing maybe, but testing is a lot of duplicate code and you can go far with just copy-pasting stuff around and building helpers.
Re:GIMP now has UI tests
by aidenialislife on Friday November 20, 2026 @12:56AM
(#50570687)
Designing for testing helps as well. Pure functions for core logic, coding against interfaces etc.
I my opinion, AI only changes the ease of writing these tests, not really the quality.
I find that the human written tests on my project were the same. We were guarding not just the business behavior but also weird cases that have nothing to do with business, and also self-evident cases that basically boil down to x=x or assert(true).
Not to mention the stupid side-effect of not updating mocks when the APIs move but tests don't test that part so they are okay for a lot of time, and suddenly after a non-related change they are not.
Re:GIMP now has UI tests
by locusofself on Friday November 20, 2026 @01:02PM
(#50579090)
I agree, some of my small projects have had an explosion of tests. It usually doesn't cause my any problems other than waiting for tests to run, but that isn't great.