It’s pretty good depending on what you are doing. With Python and JS/TS, the last checkpoint for training datasets really matters. These api’s for no reason at all to me, rot incredibly fast, so the code it generates was valid in September 2020, but now it is almost completely incompatible.

I was raised with (80s/90s); never break the public api unless you have to and make a migration path if you do; now it is ‘let’s definitely break the public api, with any minor version change and just waste everyone’s life fixing it, we don’t care’. And gpt doesn’t know this. It usually does make up code that’s close to correct and relatively easy to fix though, but you need to know how to read the docs and not panic that actually everything is spitting errors.

If you ask it for C, Java or vanilla js it fares better but there you suffer from the memory window; now with 16k tokens, we are seeing massive improvements. The api deprecation (read; we just throw away or change without warning) issue remains and is a huge issue; I cannot even see how they would properly fix that in LLMs. They are going to have outdated versions in their search space; how do they know to ignore them? And next to that there are the hallucinations.