Even being upfront about our low expectations and providing very friendly and helpful people to pair and guide, it lead to record numbers flipping the table and stomping off premises before the completion of the interview.
I don't use this approach now, as I don't think its a good use of time (and I think an interview should be a positive experience even for those that fail), but it did provide, I believe, a fairly strong signal of the attitude with which a candidate approached learning new things (but not really anything else).
Really, please, tell us more!
It's not that you're wrong, it absolutely is a really good way of finding out how someone reacts to being asked to learn something new in a stressful situation. And that is a valuable thing to know - I love working with people who get excited about an opportunity to learn something.
It's just that you have to be prepared to give so much help and the question has to be so easy because it's all completely new that you can't test competence in any meaningful way. Which means that at the end of the 90 minutes or whatever, you know exactly one thing more - their attitude to learning under pressure, but you have relatively little time with them, and you probably need to learn multiple things per session.
You also have to be careful when in the interview you throw this. A bad hit like this can throw a candidate off for the rest of the interview even if they are quite acceptable.
The Kobayashi Maru test.
1) Some people may find Fortran easier to understand than others even if they haven't touched it before. This was the case when my University decided to use Haskell* as the first language to teach students in order to level the playing field. It did help somewhat, it leveled the field for those with little and those without any prior programming knowledge. But to those who had already come to terms with recursion etc it was a very tiny stumbling block.
2) Some people need to dive deeper and gain a greater understanding of things before they feel at ease using them, but once they have reached that level, they will often outperform most other people.
* This was 20+ yrs ago and Haskell was not so well known back then.
nice to see it's more common
Oh wow, that would definitely get me sweating. All the FORTRAN i learned was 77!
* Predictive of job performance
* Either (1) difficult to optimize against or (2) easy to optimize against but in a way that doesn't reduce predictive value
* Low cost to give
* Low cost to take
How about "Google is giving tests in these 25 languages"? They could even choose languages that are purposefully dissimilar from any language that you've claimed familiarity with.
Exp: You must code in an unfamiliar language (Fortran 1988, for example) and do something simple in it. At least then you'll know that people are all starting from scratch. Time it, make sure it doesn't last more than 2 hours, see how laughably slow we all are, pick those that get the furthest.