I (kinda) solved this with neuro-lingo[0] with the concept of pinning. Basically, once you have a version of a function implementation that works, you can pin it and it won't be regenerated when it's "compiled". The alternative approach would be to have tests be the only code a developer writes, and then make LLMs generate code to match the implementation for those, running the tests to ensure it's valid.

- [0] https://github.com/eeue56/neuro-lingo