There is a huge difference between being under pressure and the unique pressure of performing in an interview. The former is being able to handle a longer term fear of failure. The latter is dealing with a short-term fear that activates the primitive brain's "fight or flight" instinct.
There is a HUGE difference, and I would never trust anything I got from the latter style interview.
All-night overtime or short deadline pressures are completely different beasts.
> Not my experience. Actually, quite the contrary.
> The single biggest difference I have ever seen (over and over again) between a good programmer and a great programmer is how they respond under pressure. Given enough time, lots of programmers can code something that works. The problem is that there are many occasions where there simply isn't enough time. Can you hit a deadline? Can you stay awake all night if you have to? Can you resolve this problem that's holding everything and everyone else up? Can you settle down that user or customer who's up in arms? Can you figure out what wrong decision was made 4 steps ago that is causing big problems now? And can you do it now?
> Understanding how well someone performs under pressure (and whether or not that makes them nervous) is hardly useless. I can't think of too many more things that are useful to know.
I'm pretty sure your examples conflate different sorts of pressure, and I don't think how you perform for one reflects how you'll perform for all. I've been in all those situations and the physiological responses are different.
There's never been a time on the job that I've felt the sort of pressure I have in an interview. In fact, if that were routine, I'm pretty sure my friends would have burned out of software development long ago. Moreover, if my job routinely required last second cowboy-style programming, which is sure to lead to bad design and bugs, I'd get a new job.
I'd say that, for most software positions, if it makes sense to hire programmers whose success depends on performing well under the same physiological conditions as are normal during an interview, something's broken in your processes.
Disclosure: As an ex-basketball player, crunch time pressure is extremely similar to interview pressure, and I handle both exceedingly well. I've always felt a little fraudulent since I come off better in interviews than some programmers who are more talented than I am.
I found the reason in the end. I wasn't extra stress sensitive. I just couldn't relax during stressed periods -- I was "on" 24/7.
Mediation solved that.
Edit: This was a bit personal, but I'll let it be. It might help someone. (AA, etc.)
Sigh.
Try as we might, IQs don't actually follow a bell curve. There are many more on the high and low ends than would be expected by looking at the curvature in the middle. Whether this is a measurement error or something else, we don't know.
Also, because of the unique position of the USA in the world (and especially places like Silicon Valley, Cambridge, Pasadena, New York) an awful lot of the right hand side of the bell curve for nations like China, India, Russia, etc. are currently living here, and often working for tech companies.
Additionally, when you do happen to get extremely brilliant people excited and working on the same thing as you in the same organization, the effects are magical. I've experienced nothing else like it.
Finally, when you're in Silicon Valley, you get the sense that there are maybe a couple thousand serious players who, if you look close, are directly involved in or are supporting most of the major efforts. It is a small world -- it seems like it's about two degrees of separation, give or take. You really get the sense that the right kind of people are the crucial ingredient that fires the whole engine of innovation here.
IQ is probably one of the less important characteristics in most positions in most companies. But it's still an ok measure for something important.
But much better to have people who are not merely good intellectual generalists (high 'g') but who are actually alarmingly good at the actual things you need to do in your company. There are as many times more of these as there are distinct things to do.
The value is defined by a bell curve.
The IQ test was invented by Alfred Binet as a way of identifying people who were having trouble in school because they were mentally slow. For this he came up with a large number of different questions that exercised the brain in various ways, and figured out how well an average kid would do. When he gave the test to a real kid he would take their performance and figure out a "mental age" that they performed at. Their intelligence quotient was then defined as 100 * (mental age) / (physical age).
The development of IQ tests aimed at adults which are defined based on a bell curve was a later innovation. The name was kept simply because it was then well-known.
Most IQ tests can't even measure above 155 or so, anyway, and they obviously can't measure below 0. So it's a little silly to talk about them being normally distributed.
That is, hearing,"You're so smart," so many times, from teachers, other students, coworkers, bosses, etc., cultivates a sense of elitism and a shock when something is actually difficult.
You're also bound by the same emotions as most other people. If you're a particularly aggressive or irritable person, imagine being surrounded by people who keep making mistakes because they're stupid. If you turn it outward, you're impatient at best and a bully at worst. If you turn it inward, you're stressed out, anxious, and depressed.
All of that said, being a professional programmer was the most satisfying thing I've done. Unfortunately, I moved on precisely because of the above paragraph -- I had managers I felt were particularly dim.
My thoughts all stemmed from the idea that compliments make people worse at what they do. At BEST they make the person continue at the same speed with a boost of moral. Imho.
We let everyone who asks take a very relaxed programming test. Simple (if you know basic algorithms, at least) problem, 4 hours deadline, work from home or in one of our free offices, programming language up to the interviewee. The delivered program is then rated by a few of our programmers who give some comments and a thumbs up or down. If thumbs are generally up, the candidate goes to the interview. The ones who get to the interview almost always end up hired.
So, basically, we have 3 tasks:
* Answer emails, send out tests, receive deliveries, and schedule interviews or say "sorry" to each candidate: This is the most stressful, so we rotate that every once in a while.
* Going over code and commenting: Enough people consider this a somewhat fun break from normal work that we don't have any problems scaling this.
* Interviewing: This has to require the R&D leader and at least one senior programmer. Since the people who get this far are almost always hired, it takes up no more of their time than it has to.
Because you have provided no mechanism to insure consistent results no matter who is conducting the assessment.
If you and I each assessed the same candidate using your methods, we could very easily come up with 2 very different assessments. And that's the trap you'll fall into as soon as you're not conducting every interview.
All 3 of your questions are important. How do you know when you have the answer?
What is the definition of "smart"? "good stuff in the past"? "works well with others"? We all have different definitions.
You may think someone is smart because they can code Algorithm X 5 different ways in 7 minutes. But I may think they're not so smart because they piss off the user trying to determine what the algorithm should do.
You may love something they have done in the past while I hate it. For a thousand different reasons.
Can they code a bubble sort in 2 different languages in 15 minutes? Can they do an iteration with an early exit, based upon an easily determined condition? How much can they speed up a piece of code with multiple obvious opportunities? Whether or not you'd ever use these particular questions, they are much more likely to give questions like yours the consistent yes/no answer you want.
I applaud your desire to avoid the sins of the past. You lay out a sensible approach. What's also needed is the application of some standards and metrics to give a higher probability of consistent results. That's all.
> If you and I each assessed the same candidate using your methods, we could very easily come up with 2 very different assessments. And that's the trap you'll fall into as soon as you're not conducting every interview.
...
> What's also needed is the application of some standards and metrics to give a higher probability of consistent results.
I don't think that's necessarily true. If you're Google or IBM, it might be, but if you're hiring for a smaller firm, it may very well be best to get someone that is a good fit for that particular culture.
Which actually sort of ties into my other post: I think your point about scaling is important to consider, but it only kicks in at a certain size, and I have a sneaking suspicion that Aaron has not handled hiring for, say, Accenture (nor would he want to).
I don't think hiring is ever going to be a one-size-fits-all, cookie-cutter, "standards and metrics" sort of business as long as you're talking about hiring people.
As many have pointed out already, there's a big difference between the pressure of an interview where you've got a couple of minutes to either win the job or completely blow it and a deadline where you usually have several days to plan and prepare.
I would also point out that if management is constantly springing short, do-or-die deadlines on workers then there's something very wrong with their planning.
This is also a good test of common sense: if they answer "will I get laid" rather than "monads", you know they don't have any.
On the other hand, this is a subject near and dear to my own heart, and while I appreciate a lot of his methods, I don't think his model scales well. In order for it to work the way he wants, he has to conduct the interviews himself.
A few thoughts...
Programming isn’t typically a job done under pressure, so seeing how people perform when nervous is pretty useless.
Not my experience. Actually, quite the contrary.
The single biggest difference I have ever seen (over and over again) between a good programmer and a great programmer is how they respond under pressure. Given enough time, lots of programmers can code something that works. The problem is that there are many occasions where there simply isn't enough time. Can you hit a deadline? Can you stay awake all night if you have to? Can you resolve this problem that's holding everything and everyone else up? Can you settle down that user or customer who's up in arms? Can you figure out what wrong decision was made 4 steps ago that is causing big problems now? And can you do it now?
Understanding how well someone performs under pressure (and whether or not that makes them nervous) is hardly useless. I can't think of too many more things that are useful to know.
So I just request a code sample and a demo and see whether it looks good.
There are too many cases where this doesn't work at all. What if they've never contributed to an open source project? (Whether or not you like this has nothing to do with their skill as a programmer and is another subject). What if they're prohibited from sharing someone else's proprietary property? What if they're bound by someone else's (illogical) shop standards? What if they salvaged someone else's shitty architecture with great workaround code? If they wrote something as part of a team, which part was theirs? How much of the analysis, design, architecture, testing, and deployment did they do? For that matter, how much of the actual programming did they do? Lots of posers share something they "worked on" but could never build in a hundred years.
To find out whether someone’s smart, I just have a casual conversation with them.
That's it? You make judgements on the fly? Using what metrics?
The idea of a learning from a casual conversation makes a lot of sense. But what is your plan? Ten people having ten conversations will come up with ten different assessments of the same candidate. Unless there is some kind of standard and a plan going into the meeting. How will you know they are smart? How will you know they can get things done? How will you know they can work with others? Without a plan to answer these questions, you're just waving your hands.
I suspect OP already has a plan of attack for his casual conversations but it's hard to say because he never says anything about it.
Ask them what they’ve been thinking about and probe them about it.
Again, this may or may not be a good litmus test. I know programmers with IQs of 160 who have written tons of brilliant code. If I asked tham what they've been thinking about, the answer could be "who will win the Notre Dame game" or "will I get laid tonight" just as easily as it can be about something truly cerebral.
Finally, I figure out whether I can work with someone just by hanging out with them for a bit.
Again, you can learn a lot about someone else this way, but sooner or later, you'll need some kind of standard and metric for this method to be replicable. Before you start, you must be able to say, "This is when I will know what I need to know." Until then this is just one guy's hand waving.
I’m amazed that so many companies use such silly hiring methods instead.
I'm amazed that so many companies use such silly methods for so many other things as well. But there are just as many who hire very well. I know many great programmers employed and still delivering great product 15 or 20 years with the same company. The same companies that seldom make bad hires. They use a lot of OP's strategies. But they also combine these strategies into a comprehensive system that works no matter who does the hiring.
I like OP's thinking. I just don't see how it will work well once his company becomes too big for him to do every interview himself. Systemitizing his methods will make it scale.