How did people who took this approach fare in your scheme? Thumbs up for solving the problem or penalised for “not a team player?”
We're hiring them to be developers first and foremost, so if that's how they work best, so be it. Then we move on to discussing their solution and if they thought it could be improved in any way. The rationale for this phase of interviewing was mainly to accomplish the following:
1) Can you take a problem statement and show progress in writing code in the language of your choice to solve it? You don't have to complete the entire exercise, but you do need to show us that given enough time, you'd probably get the right answer and that you can explain your logic.
2) Attempt to weed out the personalities who can't take negative feedback or refuse to accept guidance when they're wrong.
Most people opted to do the exercises while thinking out loud. If they paused for a few minutes or seemed lost, we'd offer hints or ask questions, etc. Let them take the discussion where ever they wanted.
All in all worked very well. Candidates seemed more comfortable than just doing whiteboard exercises and we got a really good feel for how they worked with others. We didn't really care much about the code; we wanted to see how they approached problem solving, how well they responded to feedback or questions about potential optimizations or offers of help. Or what happened if they were shown that their approach was a dead-end, but they still doggedly kept pursuing it.
It revealed personality issues very quickly. Nervousness was normal. Some people were hostile when bugs were pointed out, some simply couldn't get started even with extensive help, and others went heads down for half an hour and then came back up with the answer.
It wasn't perfect, but of all the interview techniques we used over the years, it was the one tool that we never got rid of because it proved to be so useful.