You have to know how to write useful test cases to know it works, etc

Some of the comments here actually freak me out.

I'd think you'd be positively appalled by the state of code in academia then.

It's kludge after hot patch after voodoo incantation. I once worked on a computer that under no circumstances could it be put on the web. It was running Windows 98 in ~2017 and was only used for a single program to control an instrument. I know for a fact that it's still online and running. It was slow as all get out, but no one had any kind of time, let alone gumption, let alone skill, to try to bring it up a millennium.

Most academic code is written fairly naively (it's usually the code that's written by people who have some idea and try to be "smart" that's a problem), and so can be cleaned up and made maintainable pretty easily. If AI causes more "smart" code to be written, that's going to make things worse.

On old hardware/software, that's normal, and will continue to be normal, simply because the value is in the instrument, and money spent trying to get the software/drivers to work on more modern systems would be better spent on buying a new and better instrument. The only way this would change is if we force all hardware vendors to fully open-source (and ideally upstream) their code (look at the whole IoT ecosystem).

This is a good point, using an LLM means that writing even more smart code is now unlocked.

One still needs to find that balance between smart, functional / practical and maintainable. That’s the art of coding.

People will still write a vague or incorrect description, and any AI will presumably have some error rate. You can't escape verification.