If you heavily modify them, that might work quite well. Every machine translation models contains language models for the source and target language that specify what a valid sentence in each language looks like. In NLP translation, they are approximations created by ML. Our models for programming languages are much better: We have the exact syntax, type systems should also help.

However, AFAIK modern MT systems are pretty much blackboxes and the language models cannot be separated from the rest of the model. In any case, it wouldn't be as easy as plugging a compiler into the system to check if the output compiles.

Of course. I hope you don't want to use the output, though.

Convert the tests by hand, then convert the language by NLP. Then go back and fix breakages in the test suite...