Remember, before any of this happens, I have already helped the candidate get comfortable and have made the purpose of the exercise clear: to assess where they're at and how/where they might fit in. It is not a test. Just an exercise to help us both. I offer them a soda or coffee, a little privacy, and this little problem...
You have an array. Call it "a" or whatever you want. It has a bunch of elements, numeric, alphanumeric, or whatever. You decide. Sort it. Without using a second array or a pre-existing function or routine. While I'm explaining this I'm sketching it out with my own pencil and paper. I suggest that they sketch out what they want to do themselves and then write some code (in the language being evaluated) or just pseudo code for general purposes. Be prepared to discuss whatever you want to present. Don't go nuts, just a few pages and 15 to 30 minutes. And have fun.
When I return, I have them explain how they approached it. (Here's what my code will do...) Then we go through the code line by line. At this point, it's incredibly easy for me to ask questions, such as...
Why did you name that variable that name?
Why did you use a for loop?
How else could you have done iteration?
How would you do it with 2 loops?
How would you do it with 1 loop?
Which variables are global? Which are local? Why?
Why did you reuse the variable "i" in the inner loop? (Oops)
How can you make it faster?
How could you make it clearer?
How would you change it if you knew the probability of the original order?
How would you refactor this?
How would you extend this to do...?
Which code would you put in a library for reuse?
You kinda get the picture. No 2 interviews are the same. Imagine the programmers you already know having this discussion with you and how much you'd learn about them.There are no right or wrong answers, just learning. Which is what you want.
This simple test eliminates the 90% of applicants who are not suited for this work and 100% of the posers. You can tell right away who they are.
OTOH, good programmers shine on this. It's actually fun to hang out and talk about this stuff with them. I have even had candidates email me later with revised code based on our discussion. Those are the motivated ones. Big points for that.
Having said that, I am not trying to be critical. I completely agree that having a discussion like this is extremely valuable and can let you find out a lot about a person and even eliminate most of the chaff (although I have met a couple of people who I wouldn't necessarily hire, but who could have done well in a discussion like that).
After your discussion, you might even decide you really like a person, but does your team like them? How about this, can you tell if a person can stick to a deadline? A longer discussion is necessary before this becomes apparent.
Did you find that success on this exercise tracked well with success on the job for those that you ended up hiring?
Yes. If someone didn't work out in the long run, it was rarely because of their programming skill, but for some other reason.
I understand what you're saying about not caring about the coding result, and I think you're right -- too many geeks focus too much on the brain-teaser part, and not enough on the human interaction part. But even then, it's an artificial environment, and people are nervous and stiff. Newer candidates are unpracticed. Candidates who have had a few interviews are more likely to succeed. Some small number people can bullshit their way through any interview. The best you can say about an interview process is that it might give you a chance to know if you'll absolutely hate working with someone.
My approach doesn't prove anything. It's just a tool. Like any tool, it has to be handled properly. I can immediately tell who is comfortable and who is not. I adjust for that. And this tool is never the last step, just an early and important one.
I don't learn everything about the candidate, but I learn a lot. And no one can bullshit their way through this; they have to know how to do it. Posers shouldn't waste my time; I will identify them with this every time.
It doesn't matter whether or not it's "really coding". All that matters is how effective it is in evaluating your candidate.
I have interviewed over 2,500 devloper candidates and every single one has had to code with pencil and paper, in a room alone for 15 to 30 minutes. This has always been, by far, the most effective thing I could have done.
I never cared what they actually wrote. I never once found out if it would even compile or run. But I never cared. The only purpose of the coding problem was the discussion afterward. This told me volumes.
Given a person, a discussion, a problem, and 1 to 3 pages of written code, I could ask many questions focused on a single issue and take it any direction I wanted. And learn what I needed to know about that person...
How did they attack the problem? What did they feel comfortable using? How did the deal with the situation? What kind of attitude did they have? How much did they enjoy dealing with the problem and discussing it? How well did they defend their choices? How willing were they to take criticism? How willing were they to stand up for what they believed in?
These are the things I want to know now. Not 3 months after they start working. Programming with pencil and paper and discussing afterward is the best way I've ever found to find these things out.