Looking at the demographics of the study, they only had 47 total participants, 15% of which were professionals and 62% reporting less than 5 years experience coding (which I would imagine is an underestimate assuming some people exaggerated their actual experience level). So that means they only had 6-7 participants who worked in industry and generously 18 people with more than 5 years experience. They also don’t report the breakdown of how participants did by experience. One other factor they use to measure is if the participant has security experience, but their bar for that is whether they have taken a single security class.

Given all of this, I don’t think the paper’s conclusion is convincing at all given that they were evaluating this on a pool of participants that a majority were students with not much experience programming when these tools are sold for professional use. I would bet if the study had a more uniform distribution of experience levels, you would probably see the “bugginess” trend downwards with more experience. Participants with more years programming just have had more time to learn how to use new tools more effectively in their workflows.

I definitely tweaked my methods of using Copilot plenty over the past year or so to take advantage of its strengths and avoid weak its weaknesses.