4 years ago I did this, I did this last year, last month I tried my hand at this. However, it requires the interviewer to have the skill to extract this information during the interview and recognise the slope.
Concerning asking tricky question, you can do it the Google way ( How many frogs does it take to build a space elevator ) but even they decided to stop these kind of questions or take a lot of time and find something your candidate can work on, even if the technology is known to him, seeing how fast he can appropriate himself the codebase is really a good way to judge your future new recruit potential
A short preemptive summary is: the last 10 amazing hires we made would not have been distinguishable from their online portfolios.
matasano.com/careers
We'll get on the phone and talk to you about the company and what our work looks like. At the end of this call you should have a good idea of what we do, how our hiring process works, and answers to questions about Matasano. Most importantly, you'll have a contact at Matasano to talk with and bounce questions off of through the duration of our process. We do 1-3 technical phone screens. You'll talk to a senior Matasano team member who will ask you about your technical background and talk you through scenarios and concepts from our day-to-day work. If you've been doing app security for 5 years, you'll be talking about your past projects; if you're a developer, you'll be talking about code. We do a web app challenge. Most software written within the last several years is web code. Everyone on our team needs to be able to deliver a solid web pen test. When you're ready, you'll be given an instance of a vulnerable web application and an hour or so to break it. We timebox challenges to avoid taking too much of your time. You're doing this on your own schedule, in your own comfortable setting. We do a custom protocol challenge. Every Matasano team member routinely runs into exotic network protocols. We'll throw something at you that you're unlikely to have worked with before and watch you reason your way through breaking it. This challenge seems to be everyone's favorite; candidates routinely tell us how they particularly enjoyed it. That's great! It's part of our day-to-day work here. Like the web challenge, it's timeboxed and you're doing it remote. We'll have you write a fuzzer. Everyone here writes fuzzers. We'll give you a file format. In the language of your choosing, you'll write a fuzzer for it. This gives us a chance to see how you code and to see what types of things you automate testing for. Like the other challenges, this one is time limited and you can do it remote. We've talked. We've done phone screens. We've answered questions. You've done challenges for us. At this point we both have a pretty good idea whether you'll be happy working with us. If that's the case, we'll bring you onsite for an in-person interview, which concludes our hiring process.
> And if the Y-axis is something good, depending on your definition of something good, then I think most people would pick the red trajectory over the blue trajectory (..unless you think you're going to die before you get to the crossing point).
There was an article about it making the rounds a few days/weeks ago. A guy basically takes on a class of 30+ people for two days when hiring, teaches them something, then hires the one who did best. The rest are still happy to have learned something.
Another good approach might be flat out asking "Tell me about the last time you learned something new"
1. people we have worked with before who are known quantities i.e. smart AND good
2. people with minimal experience who we deemed to be smart after talking to them for a few hours
it's worked out well. to be deadly honest nobody else wanted to work for us.
"Before I came back to academia a couple of years ago I was out doing startups. What I noticed is that when people hire they are almost always hire based on experience. They're looking for somebody's resume trying to find the person who has already done the job they want them to do three times over. That's basically hiring based on Y-intercept.
Personally I don't think that's a very good way to hire. The people who are doing the same thing over and over again often get burnt out and typically the reason they're doing the same thing over and over again is they've maxed out. They can't do anything more than that. And, in fact, typically what happens when you level off is you level off slightly above your level of competence. So in fact you're not actually doing the current job all that well.
So what I would always hire on is based on aptitude, not on experience. You know, is this person ready to do the job? They may never have done it before and have no experience in this area, but are they a smart person who can figure things out? Are they a quick learner? And I've found that's a much better way to get really effective people."