Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ...
by ezyang on Wednesday March 19, 2025 @04:03PM
(#43416650)
In fact, the test framework I was using at the time (jest) did in fact support this. But the person who had originally written the tests hadn't had the foresight to use snapshot tests for this failing test!
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ...
by diggan on Wednesday March 19, 2025 @06:34PM
(#43417957)
I don't know if your message is a continuation of the sarcasm (I feel like maybe no?), but I'm pretty sure parent's joke is that if you just change the expected values whenever the code changes, you aren't really effectively "testing" anything as much as "recording" outputs.
Why not take this a step farther and incorporate this methodology directly into your test suite? Every time you push a code change, run the new version of the code and use it to automatically update the "expected" output. That way you never have to worry about failures at all!