I don’t think anyone has figured out the proper solution yet, I personally try to get the LLM to write code similar as I would have written it myself.
If you mean asserting against hardcoded literals, then that is the preferred testing shape in most cases. And for good reason.
If you change production code and a unit test fails, that is useful. Idk why that's silly. If you want to test more complex interaction write a UITest or integration test.
I honestly think we're all living in different worlds. There's probably large differences in capability on a per-language, per-team, per-codebase, per-engineer environment that are producing wildly different perceptions in output quality.
And yes everyone works in different domains. I work at a database vendor myself, so mostly lower-level work.
I’m very happy and successful with LLMs, I just don’t think that these “just move all your tests to your type system!” or “get 100% test coverage” rules are an answer in the long term as we haven’t yet figured out how to do this well.
I personally treat tests as components that need to be well architected, maintained, and add them risks increasing technical debt and needs to be treated as such.
I was describing general strategy, that when applied in concert is more than the sum of its parts.
E.g I mentioned property-based testing, that is another angle at which a test avoids assertions that are too fine grained.
I think it has much more to do with overall architecture and design, and as for now, human oversight.
Step 3: Prompt the agent to write mutation tests to harden the suite
Step 4: Prompt the agent to write property-based tests.
Step 5: Prompt the agent for 100% code coverage
Step 6: Keep a look out for small issues with steps 2-5 above and refine a Skill or Rule to package the workflow and plug gaps in the model's current shortcomings.
Step 7: Dedicate some time to manually reviewing and thinking about interesting edge cases worth testing.
---
If you follow these steps, you will have better test coverage than 99.9% of any code an engineer ever wrote pre-AI... and it will be almost completely automated.
My team's incidents and bugs are DOWN this year while we're shipping more.
Leveraging AI effectively requires a new paradigm of interaction. The name of the game is to use techniques and strategies and guardrails that create an environment that produce good software.