Great.
Does the formal spec represent a complete and accurate description of what the code is supposed to do?
shrugs
Even if you can understand what the proof proves (not a given) do you personally understand the actual requirements for the software perfectly?
If you employed this approach, I would want to pair it at a minimum with a test suite.
Even then, how do you know that your system (covered by hill climbing suites of tests that pass), has correct behavior for all things you didn't specify?
You don't. Better to build from the ground up formally than fight with "AI" that probably got it worse than wrong first try.
I hope your pessimism is founded though, I would really like to see programming by human advance rather than the whole thing be taken over by AI. Hybrid approaches are also possible (AI in the loop but not in the core position).