> Meanwhile it has a hard time creating something it hasn't already seen, or a close variant.

How much time have you spent with the strongest models, such as GPT-4?

I've been using it on a daily basis for 6+ months now and your statement there doesn't fit my experience at all.

Indeed. I feel like people are not even talking about the same thing in here. I've been using GPT-4 daily for a couple months and it's not outputting code that "only looks correct". It is writing functional solutions that often work as is.

Just yesterday I pasted in a hastily written function and told GPT-4 to "please evaluate this function and make sure it's bulletproof". It outputted a bulleted list of criticisms (mostly true) and then spit out a version of the function that was shorter, easier to understand, and was safer.

That's not just the "next most likely token". Or if it is, then that's all any of us are doing.

It’s probably more accurate to say that what your brain is doing when writing code is more like emulating an LLM than an LLM is emulating your brain.

Converting human language to code is an ideal task for LLMs, which is part of why they are such an exciting development.

If you ask it to start explaining/extending/implementing classes/interfaces, start asking it to reason about leaky abstractions or when it’s appropriate to use Mixins, it frequently generates code that either doesn’t work or just misses the point. It’s especially hard because some of these problems actually are not tractable in the language you’re taking about due to their features, or due to limitations of your code/your knowledge it may suggest you do something slightly worse because you haven’t prompted it on factories or other design patterns that you might want to use.

I’ve had this exact experience with this topic learning python and designing my first app. GPT frequently sent my down the wrong path and wasted my time when I could of just used it to learn about a topic and make my own decisions.

It does not hand recursive thinking well at all.

Agreed. A good example of this (in Python) is to explore solutions to problems that use decorators/a more functional style vs. solutions that use classes and a more OO style. Usually both are totally good. The code GPT generates can be totally functional, yet inappropriate in context. Another good one is asking it to translate code from JS to a language without real “interfaces” and see if it uses a base class pattern or not. I review junior devs and lecture them extensively about stuff like this. (Yes, I am fun at parties.)

There are an almost unlimited number of ways you can prompt it that will produce obviously bad results.

The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results.

Right, but obviously you can see the unknown unknown problem or Dunning-Krueger popping up when software developers who don’t know everything (unlike me, of course) start using it extensively without knowing the ways in which their prompt engineering is flawed.

One of the big open questions in LLM-assisted learning is whether it helps or hinders less experienced developers.

At this point I'm pretty confident it helps them. They need to understand that it's not infallible, but I think most people figure that out pretty quickly after using it for more than a few days.

There's absolutely a skill to using these things well, and it's a difficult one to teach because it's based more on intuition that you develop over time.

The key thing is helping people understand the strengths and limitations of these tools. That's something I do think we can teach.

>The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results

Might as well just learn to code then.

My argument is that AI-assistance makes it easier and faster to learn to code.

it's not just good prompts vs bad prompts, there's luck involved, since the result is determined by dice rolls.

I used to be smug about getting good results from GPT while coworkers failed to get it to produce anything useful, but over time my luck changed, and GPT stopped nailing it. If you remember there was a bunch of conspiracy talk that GPT got lobotomized, was no longer writing full programs anymore etc, but what really happened is people who got used to rolling 7s started rolling 2s and 3s and shook their fists in anger at their change of luck.

Also, as you go down that path, you may hit it's limits on tokens/memory, and it forgets what you were originally talking about.

You can invent a programming language on the spot and it will help you write programs in it.