For the coding aspect of our interview test, I tell people they can use whatever they want to write it (LLM tools included) and they do all of the work on their own machine without us watching the screen in real time (to make it less stressful). After they’re finished, I ask them to walk me through the code, to make one addition, and fix a bug that almost everyone makes. This helps demonstrate their ability to explain code (even if they were assisted in writing it) and to modify existing code. This exercise is shockingly effective.
None of this is done as a “gotcha“. The change that I asked them to make is incredibly minor, and in most cases can be done completely additively to the existing code. As for the common bug, it is due to people “pattern matching” and misunderstanding part of the instructions, but also very easy to fix once you understand the mistake.
I’ve been very pleased with using this as a filter. In some cases, the results were surprising and, in others, they confirmed what I already thought/suspected.
The first coding tests proves they can at least string together a bit of code, explain it, and make minor changes to it. The problem solving test proves they can think critically and understand a new problem and offer solutions to it. Lastly, we have a debugging test where they have to fix existing code with a handful of bugs. Between these and the conversations we have have with applicants we have (so far) a very good success rate in hiring.
You should have not said that. They will use a model to create the prompts... :-) You should have said: I will break your code and you have to fix it in front of me.