Has anyone figured out a good way to fix shallow tests with AI? Intuitively, in webapps, there are many minute UX things folks on the team used to manually dev-test. not saying that was done consistently across all team members or always done, but it existed.

Now it's gone. Spec-driven dev isn't the answer either, some of the UX bugs areas are simply not something that ever existed in the regression suite.

I spend most of my time structuring code to be meaningfully verifiable. To me that usually means turning a large functional area into a CLI. So for example I have a pipeline of processing and enriching products for search. Thats a CLI whereas before it might be a module in my larger codebase.

Then in that CLI I have a thin front end that just parses args and calls the backend that does the work.

I test around this backend. As a kind of behavioral test.

I rarely mock, and only at some external API boundaries. I may enumerate those (they’re only 2-3 things) the tests call my code via the backend, and I can verify more easily visually the test measures something meaningful. Moreover they map cleanly to the business logic of this module.

If testing becomes brittle I can converse with my agent and show it a bug via the CLI. It can repro more easily, and with a cleaner mapping between the CLI and my tests, I can confirm it tested what I expected.

thanks. I guess the problem is more acute in larger codebases, where the code AI writes takes "inspiration" from existing human-written code (i.e. conforms to earlier human opinions), so it doesn't go out of its way to write meaningfully verifiable new code - because that would stick out like a sore-thumb in a code-review.

Writing modular, verifiable code was always an engineering tenet - but the use of AI in codebases that ignored those foundations ends up having several gaps.

For UX specifically, I have good experience with letting the agent record a demo video for me. It regularly finds issues itself while doing this and fixes them, and I get a chance to very efficiently see the UX w/o having to start a dev server and navigate somewhere and create a certain state before I can see the UX.