Okay so it generated a response which was “reasonable”

How do you know it was correct? Because you checked it’s entire output manually and determined it probably wasn’t too wrong?

So what happens if you now trust it to write firmware for some difficult old timey hardware that nobody understands anymore. It seems correct. But then it actually was just making it up and the coolant system of the power plant breaks and kills 20,000 people.

Are you assuming occasional non-factual output is going to be an issue in the future?

You certainly have to validate the output but I am pretty sure not too far in the future AI will be able to do a better job than humans.

> So what happens if you now trust it to write firmware for some difficult old timey hardware that nobody understands anymore.

How well would anyone do?

Would you trust me?

You should see who they hire to write firmware. I wouldn’t trust them with my cable box.

Exactly

Hire me

By trying to run it usually. It is sometimes wrong, and I amend things. But I’ve had more occasions where I thought I was right and it was wrong and after a long debugging I realized I had failed to grok some edge in the language and it was indeed correct and I learned something new.

But I would suggest not using a LLM to make nuclear reactor control system code, just like Java.