Indeed. At this point, one thing I'd do is stick a candidate in front of some code that (a) didn't work, (b) which came from ChatGPT, and (c) which ChatGPT cannot itself fix, and see if the candidate can fix it.
But maybe they will still use chatgpt to help them figure out the solution. A non-trivial number of my interactions with it are of this form, "no, that isn't right, fix this part and re-do the rest". And that should be fine. Or it should be fine to not do it that way.
The goal is to get a sense for how the person you may or may not be working with approaches solving problems. Sometimes LLMs are part of how they do that, sometimes they aren't. And you can learn something about them from that, either way.
Ok, then that seems like a pretty reasonable thing to assess?
> I'd rather have a developer on my team that demonstrates high technical competence than one that is GPT-skilled, but doesn't know what questions to ask GPT nor how to judge its responses.
But "what questions does the candidate ask an LLM and how do they judge its responses" is part of the interview, if you don't forbid them from using an LLM!
Now, if they don't want to use these tools, if that's not part of their normal process while working, then that's totally fine too. But if they're comfortable with these tools, if they are part of their normal set of things they use for their work, then you're doing yourself a disservice by designing an interview process that is incapable of accomodating that.
As the article shows, it's much easier to mimick competence with the help of a chatbot. That obviously doesn't mean one actually is competent to produce good work in a real setting.
I don't think that's what the article shows. I think it shows that it's useless to ask "leetcode" questions and focus on the code produced rather than expecting candidates to walk through their thought process and show what tools they're using to aid it.
In that case, let everyone use ChatGPT and similar tools.
Those that know, will likely not use it much. Those that do now know, or are not too confident on themselves, will use it more.
Your ability to use ChatGPT effectivelly is highly dependent on your technical competence.
The interview is meant to measure your acquired competence, because this is the harder part. Learning to leverage that competence using ChatGPT is very easy.
I'd rather have a developer on my team that demonstrates high technical competence than one that is GPT-skilled, but doesn't know what questions to ask GPT nor how to judge its responses.