Sounds a lot like model training and I’ve treated this sort of programming with AI exactly like that importantly making sure I have a test/train split

Make sure there’s a holdout the agent can’t see that it’s measured against. (And make sure it doesn’t cheat)

https://softwaredoug.com/blog/2026/01/17/ai-coding-needs-tes...