I fix this by manually prompting many small changes (with a very fast model — smaller models are fine for simple changes / additions, and you can iterate fast).

So I can still work way faster than programming manually, but I stay in flow. And most importantly, my mental model stays synced the whole time. There's no catching up to do.