Because the computer in this case is relying on a vast repository of human knowledge encoded in these languages to go from 1 to 3. Machine code was probably not in the training set.

Would be interesting if you could train a GPT-N model to be a compiler somehow.

True of GTP and other LLMs, but an AI trained on machine code could simply output machine code. The difficulty would be that it wouldn’t then understand natural language prompts.

Why not have it be human readable as well? Good to be redundant. They can automate the compilation, and we can't always rely on the machine

Because we don't have a system that understands language and generates code. We have a token predictor trained on a huge corpus of code mixed with explanatory language. And the existing huge corpus of code mixed with explanatory language consists, necessarily, almost entirely of code written in human-oriented languages.

Interesting idea. The verifying correctness kind of seems like the most important part though, and entirely non trivial. Test driven development maybe?

It seems like we might look back at writing code in 15-20 years the way we now look back at something like punch card programming

For auditing the generated instructions, as auditing machine language itself is unrealistic.

Step 2 is more token efficient, I suspect we'll not get rid of it.

If we eliminated step 2 and achieved equivalency with or even slightly improved upon the state of the art, we would still be at a significant disadvantage.

The problem is that the current best (publicly known) results are nowhere near good enough not to be verified by a human prior to deployment. So if we get to _that_ point having skipped step 2, the verification will have to be done on machine language.

And since there's no human-abstracted code intermediate between the spoken language and the assembly, the algorithmically-searched-out analogies of language to purpose are fairly likely to baffle everyone.