That's not what is happening.
Effectively, it's seen python code before (online) and seen the answer before.
There's no understanding.
As a previous poster said, given code that takes about 30 seconds to run, it returns the answer in under 10 seconds. It's not running the code. It's guessing the answer.
Also, it does not matter that the code takes 30 seconds to execute — I’m not claiming that the model is executing the code. I’m suggesting that the model could be understanding what the code is meant to do, and producing the output from that understanding. The code’s inefficiency is irrelevant in that process.
Sure someone must have posted online about ls and rm, but a jumbled combination of those where the model is able to keep track of files listed, created and deleted in the specific context of the chat? Mindblowing!
I think the biggest mistake we're making these days is that we somehow believe our intelligence is different from any other form of intelligence we might invent.
I'm so grateful that I got to witness the rise of the Internet & PCs, the Doom to Unreal Engine journey, and the transformer revolution, and that I'll get to see the 100x version of GPT.
It's looking at the code, and comparing it to what it's seen online, and what code that looked kindof like that output.
That is exactly what you are doing when you glance at code before stepping through it in your mind.
Or have it design a whole SQL schema for you, then ask it to make it work in MongoDB.
The progress in this area is insane. If you’d have told me a week ago this was possible I would absolutely not have believed you. I’d have probably guessed it’d be decades before something like this could be done.