nsaftarli

Also, their classifier just uses manual features rather than doing any sort of meaningful analysis. As in, the input to their model is a tuple of ~20 features consisting of things like, whether the article contains the word "but", or the character "?".

All of this is easily fixable by using better prompts. It's 99% effective for a small set of articles from one journal (sampling not specified), assuming that the synthetic samples are created with the minimum level of effort using GPT-3.5.

It's not that their methods are wrong, it's just that they're absolutely useless for anything beyond an extremely narrow range of data. In the ML field this is called overfitting.

So, not 50% (which I think we all knew), but an estimated 40% of the people using Copilot (some percentage of devs), with that figure being heavily dependent on how the study was done. Are people self reporting these figures? Because you'd have to think that there'd be a sampling bias there.

Anecdotally, copilot has gotten in my way more than it has been useful, so I'm having trouble buying these figures.