This reminds me a bit of the some of the problem solving experiments done comparing pet dogs to undomesticated wolves. The experiment wasn't overly complicated, a piece of meat was placed in a locked cage and the animals were given free reign to try their paws at getting to the food. The wolves were persistent, some employed tools like sticks to push the food out, some managed to navigate the clasping mechanism on the cage. The domesticated dogs more often than not quickly gave up and looked to the human present for help.

Sounds like dogs found another way to solve their problems ie get humans to do it for them. It's a pretty good solution as long as there is a sympathetic human around. Of course, they would be SOL if they ever find themselves alone out in the wild.

I'm not familiar with the study, I wonder if they also did the test with no human around. Anecdotally, my dogs can be pretty lazy about getting into things when I'm around, but seem to get creative when I'm out.

Think you're referring to Frank & Frank 1985. It's pretty neat in terms of the tests and responses. Lots of springs, levers, and other puzzle boxes with the wolves almost always eventually solving the puzzles unless the wolf was nervous or agitated by the situation for some reason.

"The wolves generally attacked each puzzle immediately upon release from the start box and persisted until either the problem was solved or time had run out. In contrast, the malamutes investigated puzzle boxes only until they discovered that the food was not easily accessible, after which they typically returned to the start box and performed a variety of solicitation and begging gestures toward Experimenter 1."

There's a couple others that have followed on that are kind of neat to and related. Marshall-Pescini, et al. looked at wolves and dogs ability to play shell games, recognize hidden food choices, and rationalize about whether risky choices that don't pay out are better than somewhat not preferable food pellets. Part of the result from that test was that dogs may just not care as much as wolves. That wolves with their diet and carnivore nature, have a much stronger preference toward what researchers believe is the preferable choice. Yet, from the wild dog perspective, they took a long time to have any testing preference for meat vs pellets, compared to the wolves immediate preference. [2]

Which is actually vaguely related to the topic article. You get a mediocre food pellet, but it solves the task, so you don't really care that much and move on. The "quality" of the food pellet in modern human existence has limited bearing.

[1] Frank & Frank 1985, "Comparative Manipulation Test Performance in Ten Week Old Wolves and Malamutes", https://www.researchgate.net/profile/Harry-Frank-2/publicati...

[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4993792/

Some of the dog breeds, most notably French Bulldogs, have been specifically bred for qualities that enhance their interaction and dependency on humans (like having a unique set of facial features and expressions that many people find irresistibly cute and compelling).

> bred for qualities that enhance their interaction and dependency on humans

That's basically the whole idea behind dogs.

> OTOH, the cost to this progress not having been earned is pretty high, too: He takes a lot of AI-generated boilerplate for granted now without understanding it or the concepts behind it, so when the AI gets it wrong or forgets it, he is unable to notice what's missing.

How materially different is this from "copy-pasted boilerplate from an example on a website that isn't fully understood"?

I've personally found one of the biggest advantages for learning a new stack with ChatGPT is being able to say "hey, how do I modify this boilerplate for [specific piece of functionality]" or "hey, I have this code and I'm getting this error, what should I try" vs just trying to find websites with other examples of slightly-different boilerplate or trying to start from square 1 (which would often mean dedicating days or weeks to less-immediately-relevant tutorial foundation projects).

> How materially different is this from "copy-pasted boilerplate from an example on a website that isn't fully understood"?

When finding and copying, you have to employ at least some degree of critical thinking. The result is rarely the first on the search results, and usually cannot be used without some adaptation. Generated solutions usually require less adaptation.

> When finding and copying, you have to employ at least some degree of critical thinking.

You and I have workers with vastly different frontend "engineers" in that case, especially in web agency shops where "faster implemented === better" in most cases.

Haha. But taking the statement seriously: Do you think the engineers in question employ more of their critical thinking capacity when using code from a generator than when adapting existing snippets?

>> He takes a lot of AI-generated boilerplate for granted now without understanding it or the concepts behind it i have seen this too in junior programmers. It is indeed bad but than i think of myself without internet and in some ways i too become handicapped like them unable to make meaningful progress without access to documentation so its a mixed bag. Good programmers will learn to identify gaps and filling them so that they can better utilize the tools others would be left behind.

For senior programmers its an absolute productivity boost.

Yeah, agreed - if you know what you want, and can write the reqs to coax it out of the model, it's a great typing accelerator.

> it's a great typing accelerator

It's great at this, and I use it like this a lot.

Like I know how to write the code, and exactly how I want it to be, but I faster describe that in words and let GPT4 write it out like I want it to, than to write it out myself. Even in the cases when the code isn't even boilerplate.

Sometimes the system prompts need a bit of tuning, but time spent on that tends to even out after some usage.

That's really interesting.

You guys should write an article about it covering both perspectives.

Recently my ex-employer said he fired a junior dev because he was relying too much on paid ChatGPT, without understanding the concepts.

This is exactly why using GPT to supplement but not supplant writing code is very good. Code generation is generally more helpful as you go to higher layers of abstraction as long as you can stay at that layer of abstraction. Once you need to jump a level lower because your abstraction fails in a specific context you need pre-existing knowledge of how the abstraction works in order to ask GPT the right questions.

Hence, depending on the domain knowledge of the piece of code you are dealing with, GPT can be very helpful in generating a good scaffolding to get off the ground quickly (i.e you want to write a web app but don't want to deal with having to learn how to write a whole react app etc.), but asking GPT to...write an optimizer for a C runtime would end up with poor results as its heavily bent on the specifics of that task where a specialists' knowledge would outweigh any abstractive advantages.

One very useful experiment I did early on was try to solve a problem with GPT where I had deep domain expertise and see where the cracks are vs in one in which I had very poor expertise. This led me to make my abstraction based statement above, and so far I've seen it remain true with every successive version.

How do you know he’s progressing faster than otherwise? At least, for me, the yardsticks I have for how long it takes to learn to code are: (my own) self-learning experience, (my own) experience learning in the classroom, and then working with new students. If I mentored somebody one-on-one I guess I’d probably be surprised at how fast they learned.

I have some prior experience with mentoring beginners and juniors (I've done Google Summer of Code mentoring four times, participate in KDE's own mentor program, same at $dayjob, etc.). Of course not all students are created equal in the first place, so you have to trust my gut a bit in terms of compensating for that.

The velocity of task completion on tasks that are within reach of what he can figure out by "pair-programming" with the AI is very high. However, the failure modes are devastating - when he gets stuck, he gets stuck completely, which no idea what to do next. And ChatGPT can't assist with the overall development plan, or at least it's a lot harder to ask it about it. Some questions are difficult to ask without the hindsight afforded by experience.

With earlier students, pre-AI, the work got done more slowly but afforded many more little mental on-ramps for "what to do next", or at least ideas. Partly because the ability to read and browse code gets trained much more if you have to piece your solutions together via reference code and docs, vs. getting code handed to you by gen AI. If you can read/navigate a codebase more effectively, you are also more likely to be able to generate ideas what to touch next and why. Partly also because your muscle for trying things out and experimenting gets trained more if that's your only choice.

In sum, as a mentor, when a student gets stuck I usually have more to work with in the dialog that follows. Ideas to interrogate, experiments to brainstorm, assumptions to challenge. With the ChatGPT-assistent student, almost nada - I've "caught" (this is of course perfectly fair under our agreement) him leaning on ChatGPT to even have the convo with me, handing my messages/questions to the model and coming back with what it generated, asking me whether ChatGPT got it right or not. I wind up being the second opinion that corrects/checks the AI, not the student, who is mentally fairly disengaged from the process by that point.

What I'm getting out of this experiment is an idea of what kind of guidance I will need to give future mentees on how to use the AI tools appropriately for their own development.

That’s really interesting. You seem to be in a good position to make some useful observations.

My last semester working with students was a year ago, and we were aware that they were going to ChatGPT for things, but not really sure how to deal with it. It seems obvious that in the future these tools will play a part, but of course those of us who learned without them aren’t in a particularly good position to teach how to use them or to structure things around them. It is a temporary problem but a pretty big one, IMO.

I wonder if a school-sponsored GPT with monitoring from the teaching assistants could be part of the puzzle; it seems really neat: it sets the expectation more realistically (some AI tools will be used whatever the policy is, may as well be ours), and gives the teaching staff some insight into how the students are using it and what they are struggling with. Although, it would have to be a pretty state of the art model, you’d want the students to prefer it to their own… also, setting the expectations correctly (it isn’t authoritative, it is on you to double check it—awkward, for a school-provided tool).

Anyway, hopefully there are more folks out there like you, actively experimenting with this stuff.

Seems obvious to me that using generative ai to learn coding would be akin to going to the gym and using hydraulic machinery to lift the weights. You get it done, but get no benefits out of it.

I was reading over a student's code and he couldn't tell me his own variable names, couldn't even find one when I referred to it conceptually ("Where is the variable that holds your cache?".... no answer at all, not even a guess. Just, "I don't know.")

Having used GPT all year myself, I quit using it to generate new code for me for the most part. Back to StackOverflow, books, and of course reading boring documentation/manuals. I'm not closed off to the idea, just that I'm very worried it could atrophy certain skills.

I’ve done something similar lately, partly I started because I don't always have internet. But I also started to see the benefits of using manuals and conjure up things on my own.

It definitely feels harder to do, and doing small tweaks here and there is frustrating as you have to really understand everything. I however find it quite rewarding because I feel I become much more capable as a programmer, and I rely less on third party dependencies for everything. I come up with more original solutions and become more able to combine tools to solve different problems.

That's not surprising.

What would you expect from pupils if the teacher gave them all solutions when they ask?