Great.
Does the formal spec represent a complete and accurate description of what the code is supposed to do?
shrugs
Even if you can understand what the proof proves (not a given) do you personally understand the actual requirements for the software perfectly?
If you employed this approach, I would want to pair it at a minimum with a test suite.
Even then, how do you know that your system (covered by hill climbing suites of tests that pass), has correct behavior for all things you didn't specify?
You don't. Better to build from the ground up formally than fight with "AI" that probably got it worse than wrong first try.
I hope your pessimism is founded though, I would really like to see programming by human advance rather than the whole thing be taken over by AI. Hybrid approaches are also possible (AI in the loop but not in the core position).
Even today, in limited well-defined problem spaces, there are sometimes good algorithms for this sort of thing. See Dana Angluin's algorithm for learning the DFA that recognizes a regular language given through an interactive process of having the user provide examples/counterexamples, which serve to provide an increasingly constrained spec.
Currently, a lot of instances of the first situation get the treatment befitting the second. This is because it's still too hard to use these tools at scale. However, the trend is certainly towards improving tooling and the cost of doing this will go down a lot.
Maybe in a hundred years programming will be replaced by either "formally specify what you want", if you know what you want, (AI fills in implementation of how it does what you want, as well as a proof that it does) or, if you only very roughly know what you want, explain it to the chatbot and answer it's questions.