(Of course, in a real interview I'd have a good answer for what you wanted to hear. I'm just saying that an honest answer would be useless.)
It also biases you against people with many small accomplishments as opposed to one grand one. But people who can handle 20 different small tasks on a schedule are also valuable. Your organization probably needs more of them than me (I'm the one big thing sort of person).
When you think about it, the fact that we almost universally hire candidates based on their narratives sounds fucking insane.
I've managed a team and there are those who will find a grand solution to a thorny problem or find a solution to something you didn't think of as a problem and there are others who will plow on to fix countless bugs and make the system better for it. Both are needed, and others as well along the spectrum.
I believe that knowing your strengths and playing to them would be a way to get the best for both sides. Unless of course the interviewee is looking to change course completely.
That actually tests a set of skills, which puts it well ahead of various interview questions which test nothing of relevance. But it's also a limited set of skills, some of which a recent graduate might reasonably lack but might be reasonably be expected to obtain quickly (if the right kind of smart, committed, a quick study etc).
The problem is I don't know really know how to test for anything but current ability to write reasonable code for a not-too-large problem. The more experience I get interviewing, the more decisions feel arbitrary to me. And I'm pretty skeptical when other people claim to have it figured out.
I think it is a fantastic question to ask someone out of college. It shows those people who are passionate about something and went beyond the minimal requirements for their degree to achieve it. A passionate student could answer this question in a number of ways; he could point out an open source project that he published on Github that has users, some research that he did with a professor that got published, or even a class project that got him interested in the current position he is applying to. I mean, John Resig would talk about the projects he did as an undergraduate that lead to jQuery[1].
If you dig into the details of a candidate's answers to this question, you can find skills that they might have forgotten to put on their resume. Such as the fact that they know git or Matlab. You can ask them whether they used design patterns in their project and use that as a launching point for a short quiz on design patterns.
You do lose a bit of control of the interview process when you ask an open-ended question like this. A candidate can go off on a tangent about something irrelevant to the job. That is a risk. But it could turn out that you learn more about the candidate from his answers to this question than the puzzle problems that you might have planned to give him.
"Passion" doesn't matter.
What matters in a job is effectiveness and competence. Effectiveness gets things done. Competence ensures that what's getting done isn't going to backfire and create more work down the line.
In our common usage, "passion" describes the coder who stays until 9:00PM on a Friday hacking on things. But that property doesn't tell a team anything about whether that coder is the most effective use of headcount dollars. Passionate coders stay late working on compilers for obscure languages, or rewriting ugly but working code into trendy new languages and frameworks. Even if they're building important capabilities for a team's core product, they're doing it in a way that is basically impossible to schedule, on timelines that are impossible to reason about, because "passion" is idiosyncratic.
Passion (about what? quality? new technology? learning? building things? these are all different things) doesn't hurt anything. It's not a bad thing. It might even be a good thing. But it is not a crucial attribute of a new hire on most teams. If you need passion, consider whether you're hiring cofounders, not teammates.
This is to say nothing of my total lack of faith in the ability of engineers to assess "passion". For any given engineer, candidates that happen to do things in their spare time that the interviewing engineer happens to enjoy or appreciate are "passionate". Meanwhile, a supremely competent professional who, given a spec, can reliably estimate the time to execute that spec and then actually nail that schedule, that person is a 9-5er, because they don't have strong opinions about Node.js.
•Shudder•.
Almost every idea that engineers have about hiring is broken.
You're right, "passion" isn't one of the things to hire for. At best, in my experience, it's highly correlated with the breadth of experience that fuels "creativity" and "effectiveness" - at least, in the sense that it causes people to be persistent against hard and unclear problems. It's fair to ask whether this should be required for teammates or just co-founders -- but even in my most enterprise-y gigs, I've rarely seen a team that could succeed without people with passion helping drive the work.
Maturity is also critical. And passion and maturity are not at all mutually exclusive.
If there are misconceptions about passion floating around the memespace, then you are certainly right to challenge them, but let's not throw the baby out with the bathwater.
As bad as management's actions were for me, the absolute worst thing was watching the change in this guy. Before, he would constantly seek out every way to improve the customer experience and go above and beyond. He turned into a guy who literally said "I don't give a shit."
Almost every person I've ever worked and every job with has been passionate about doing their best, unless actively stopped from doing it. The one or two who didn't could not have been filtered out by any kind of interview question because they knew how to bluff their way through the process. But you can tell after working with them for a few months that they just try to do the minimum necessary to not get fired.
An attempt to recruit for passion is a bad sign. You can't recruit for good morale, either.[1] Either your company is a nice place to work or it isn't. Either your company will bring out the passion that almost all developers have or it will destroy it.
[1] Yes, there are assholes who destroy morale, just like there are total clock-punchers. Sometimes you can tell by having a normal human conversation, and a normal human conversation should of course be part of any interview.
If you are are trying to find the liars, assholes, frauds, and professional slackers, you need to realize that
1) they have you vastly outmatched in this contest unless maybe you are bringing in psychologists trained in this,
2) there is a social cost from treating your prospective candidates as if they are one of these things.
I think it's because building great products isn't just a matter of execution. It's also, to some extent, an art.
I have trouble imagining someone being effective and competent with absolutely no passion. I will grant, though, that there are competent people who are more passionate, and those who are less so. I think a successful organization could have a blend of people across the passion scale, but I don't think one could build a successful product business without at least a few people who are passionate about what they're doing. Maybe in other kinds of businesses you could; I don't know.
Of course it gets trickier when the more competent one seems the less passionate. Then it's a question of how much to weight the two dimensions (along with whatever others one is considering). I have no magic formula here; I think it depends on the circumstances.
There seems to be a subtext here that work sample tests give no indication of passion. I don't think that's true; when I've given people programming exercises, I think the craftsmanship of the code (is it well-modularized, well-documented, nicely indented, etc.) is indicative of their passion. Competence, to me, is more about whether the code works.
So yes, I'm using craftsmanship as a proxy for passion, so I suppose next you'll ask, why not just go for craftsmanship? The point we seem to be coming to is that passion is difficult or impossible to measure directly, so why not stick to qualities that are more measurable?
I don't think everything important is easy to measure.
Thus, the only way to be effective and competent is to have a passion for what you do. Otherwise, if you limit yourself to 9 to 5, you'll be treated as such. And it works like that in many industries, and in those industries, those with a passion will routinely do better, whereas those that don't, will stick to the 9 to 5 places.
Almost every idea anyone has about interviewing is wrong.
Interviews are a terrible way to recruit. It's amazing to me that STEM companies with money and scientists still use interviewing in recruitment processes and haven't found something else.
There is the "on the job interview" where a candidate is invited to do the actual job for a day or half day. For obvious reasons this method tends to be used for minimum wage positions, or for places where you're recruiting people who you know might interview poorly. But it seems to be a great way to recruit people.
Even if they're building important capabilities for a team's core product, they're doing it in a way that is basically impossible to schedule, on timelines that are impossible to reason about, because "passion" is idiosyncratic.
Yeah, passion is a bit of an x-factor, isn't it? By itself, it is meaningless. You may get something unexpectedly great or unexpectedly bad. But, using myself as an example again, I was able to get a project at work done well ahead of schedule because I had already been working on something related in my spare time.
Passion (about what? quality? new technology? learning? building things? these are all different things) doesn't hurt anything. It's not a bad thing. It might even be a good thing. But it is not a crucial attribute of a new hire on most teams. If you need passion, consider whether you're hiring cofounders, not teammates.
Inasmuch as someone is proud of a project they did and can talk enthusiastically about it, and that project is related to work currently going on at the company they are applying to, I think a passionate candidate can be better long term than another candidate that is initially more competent and more effective. If experience is the main reason that candidate A is more competent and effective than candidate B, then candidate B may eventually surpass candidate A as a better employee simply because passion drives candidate B to keep honing his skills in his free time. I think this is especially true for college graduates or other junior candidates who may not have anything else to go by but their projects.
These skills can come with experience, but can also be personal traits. An employee that doesn't code the week-end but focuses on his job while on site and takes pride in being professional and reliable is a solid asset for any company. In the long term he might be a corner stone of your team, balancing the ones that are half dead because they didn't sleep at night to work on the projects they really invest themselves into.
whereas others cap at, "well, I did manage to build the world's first and largest bitcoin currency exchange and reach volumes of about a million a day before we had to shut down."
when you find out they did it by themselves, that is a huge difference. like two orders of magnitude FTE's.
it's like, if some guy spearheaded and came up with the whole idea behind the new Mac Pro, in an era of stagnant desktops, the whole mechanics and central idea (with the heat core). This question lets you know this.
a million examples, this is really just one.
The big issue is, what does this style of interviewing do for you with a real-world candidate selection challenge --- say, 3 developers, all similar on paper?
There is no reason the best candidate necessarily has the best story. The story of your "best accomplishment" has as much to do with random chance as it does with capability. This question demands that an interview somehow de-confound the luck element and the skill element.
And to what end? Even if you got a clean reading on this question, somehow equalized between the candidates, I ask again: do you really know a lot more about how the candidate will actually perform on the job? Because that is almost the only question that matters in candidate selection.
"Gee whiz, I'd say my biggest flaw is that I'm TOO passionate about technology!"
Maybe I coded an iphone app. Meanwhile you coded an iphone app, and fixed bugs in a huge legacy codebase, and did hiring and code reviews and mentored new employees, and contributed to open source projects, and introduced your company to source control and automated testing, and masterminded a trivial code change that made the company a million dollars a year, and you spend your free time running a programming club at a local orphanage.
If we're competing for a job and we're asked about our single biggest achievement, we'll both say iphone app. An employer who wants to accurately gauge who is the better employee should probably dig deeper.
Say I saved a million dollars a year by adjusting a configuration setting, no coding required. That might go over well with a business type, but if the interviewer is all about the programming, it doesn't demonstrate technical mastery. There will be no scope for talking about the technical challenges, the languages and databases I used, what I learned from the experience, or how much I enjoyed it and grew as a person and a professional.
Say I introduced source control and automated testing at a medium sized company. Again, the technical work is limited - you can install Jenkins with apt-get, the real challenges are political and educational, changing 'official' workflows and 'supported' software and convincing people to sink time into using the tools for long enough that they can see the benefits which are not immediately apparent. If I'm being interviewed by a young guy at a startup they might think they have no politics so my skills will be useless - or worse that I'll introduce politics. Or it just won't seem like much of an achievement, after all the six of us at this startup switched from bzr to git in an afternoon.
Needless to say, if the original job description says "wanted: someone to teach us source control and automated testing" that would be a different matter :)
Though this question can be a testimony to see whether the candidate is articulate at describing things. People who have trouble to describe problems to another person is usually not a good co-worker to have (no matter how competent he or she is). You will just waste so much time going to put his or her words into complete picture. or the new co-worker will work on his own problem the whole day and no team work.
What you need going into this interview is a solid understanding of your corporate culture -- what behaviors you value. Maybe you work for a bank, and the most important things for you are integrity and conservatism. Or, maybe you work for an early stage start-up, and calculated risk-taking and team spirit are most important.
In either case, as you walk through these questions, you ask about the how's and why's of the person's behavior. Doing so is the best way to figure out if the person is going to act in ways that you would want and expect them to. Again, the significance of the biggest thing itself is small, it's how and why they achieved it, gleaned through the 20-odd follow-up questions, that will get you to the answer.
You mention college hiring -- yes, that's a different beast. A lot of companies have success with paid internships the summer before graduation. You can also ask some of these behavioral questions, but unfortunately our educational system doesn't include enough collaborative, project-based work to get a good baseline. Nonetheless, a college student who has done significant summer project work, or who has worked on an extracurricular project, should have some data to extract.
Job interviews suck, and I feel like people who have been hiring in this industry for a long time recognize at some level that they're essentially random functions, made workable only by dev teams willingness to put up with employee churn and extreme inefficiency.
I don't agree that this question is a silver bullet, but I think it's valuable to see what sort of achievment the candidate considers significant, to ask follow-up questions about how they handled problems along the way to that achievment, etc.
The question helps reduce the power assymetry of the interview, by letting the candidate choose something they're interested in talking about and presumably familiar with. I find that people seem more relaxed when they get into the flow of describing something they're interested in, which lets me explore various aspects of their working style with as little stress to them as possible.
Job interviews do suck, but the alternatives I've seen proposed suck at least as bad, though in different ways.
Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE).
We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too.
Some other things to know about that guy:
* No prior crypto experience
* No prior software security experience
* Less than 2.5 years dev experience across both jobs
* From flyover country
* Physically uncomfortable during the interview (something noted by interviewers on his feedback)
How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess.
Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf.
Yes, I think someone is going to make a lot of money correcting that market inefficiency.
† If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.
I agree that work sample tests can be quite valuable.
Job interviews are hostile. In fact, they're one of the most hostile experiences most of us have to deal with. I watch candidates in face-to-face interviews very carefully now, because this is a problem I am studying, and when you stop and consider candidates not as gamepieces but instead as human beings, you'll be amazed at what you notice. For instance, a majority of the candidates I've sat down with in interviews have, subtly but noticeably, been physically shaking.
Job interviews are like a game of Jeopardy! where one of the players is being filmed and broadcast on live television and the other player is playing from their couch and has been given an answer book, and the poor televised shmoe is judged on the same curve as the person on the couch.
If I as someone interviewing for a QA role ask them a question such as 'Tell me about the QA work you are most proud of.' And they spend the next 20 minutes not talking about anything QA related it's a red flag. And an important data point to me when doing a retro with my peers. Perhaps that recent scenario is a reflection of the folks my company is able to attract in this area.
Conversely, if I am interviewing with you and I ask you, what's the most interesting thing your team has done and how did you help them accomplish it.' And you go into great detail about a deep area in your field that's part of the product your team works on its useful and important information when I need to make a decision. However if you are only interested and talking about me, my technical (or lack of - I know very little about security) understanding and you brush off conversational attempts to gain information that will help me judge the book by its cover it's usefully info when I need to make a decision.
I am not sure why your tone was so hostile? I did not mean to imply anything by my question. I am merely looking for other ways that people have found works for them.
Second, you're assuming that the way a candidate interacts with your team in an interview is an accurate representation of how they'll gel with the team down the road. Some of the shyest people I've interviewed have gone on to be the ones I actually end up talking to all day. Part of that is because interviews are hostile and make people shy, and part of it is that different people take different amounts of time to open up to strangers.
Finally, you're describing the "gut check" interview process used by almost every engineering team since the 1990s. We have evidence for how well that process works. Answer: not well.
As an aside: the idea of hiring candidates based on how well team members "get along" with them is repellant to me, as is "culture fit", since they're basically ways of making sure you have a team full of people who listen to roughly the same music, drink roughly the same beer, play roughly the same X360 games, and like roughly the same technology stacks. But that's an emotional argument; the real argument is that it doesn't work.
I think the problem is that everyone on HN thinks things are binary, when practically every scenario in this universe is not. You can't just eschew potential team fit because you want a rigorous hiring process that depends only on candidate qualification. The opposite holds true as well. Here are the two things you need for an awesome team:
1. A team that works well with each other.
2. Individuals on the team can do their job.
>>> the real argument is that it doesn't work.
It's not an argument, it's a statement. It would become argument if it would be supported with something that is not of equal weight with the contrary claim "no, it actually works quite well". Everybody has their anecdotes, do you have more than that?
Is your argument here that conventional engineering job interviews...
(the sort where candidates interview with a series of engineers from the team all of whom ask different sets of questions and seek to gauge from their answers previous accomplishments, working style, technical competence, and team compatibility)
... are generally an effective way for teams to screen candidates?
Or, ask an engineer how many times over the course of their career they've worked with "worthless" teammates, or people who should never have been hired. I've worked across a decent slice of the industry these past 19 years, and from what I can tell, this story is universal.
Another strong piece of evidence on my side is software quality, or the pervasive lack thereof.
Another piece of evidence is fizzbuzz.
Another piece of evidence is the (harmful) trend of demanding that candidates work on a contract-to-permanent basis as part of the recruiting process; what is this other than an attempt at work-sample testing that also happens to exploit candidates and repel the best of them?
I agree with you that close to 100% of working engineers are hired this way. But that doesn't prove anything; it's just as likely, perhaps more likely given the evidence, that what it shows is that development teams can ship marketable software despite terrible inefficiencies.
If you think that conventional hiring does a good job of selecting effective and competent programmers to the median ISV (and note here I'm stipulating ISVs, the best-case scenario for your argument, not line-of-business software teams at F500s), fine. I don't feel like I know a lot of smart engineering managers who are comfortable with the predictive power of their interview systems.
I deeply respect you for this and many other posts. This was a really good point. I hope i can run into you at blackhat or RSA this year.
What I like to ask is what project or achievement people are most proud of. That forces them to use their own subjective judgment rather than attempt to cast about for some sort of objective definition of "significant".
There are many reasons why I love this question.
First off, it puts things squarely into the realm of the real instead of the abstract. It can tell you a lot about whether or not you're dealing with someone who gets things done or just someone who "does work".
From there it's a quick jumping off point into the specific skills and competencies of the candidate, viewed through the lens of productive work. It's a way to see what skills people have in sort of a sideways way, because you see what skills they actually use, rather than the skills they tell you they have.
Perhaps most importantly, it tells you about a person's priorities and values. It tells you the sort of things they think are important. Did they do something that actually delivered value to anyone? Did they do something that required overcoming obstacles or did they just cruise along and build something "cool"? Or something that was technologically sophisticated but useless in practical terms? Do they value titles and money over real accomplishments where something of lasting importance or value is built and delivered to users?
It also gives an opportunity to explore a lot of issues that are hard to suss out otherwise. For example, does the candidate have initiative or do they just follow along? Do they strive to improve the way things work or just tolerate the status quo? That can be evidenced if the project they describe was something they took on of their own accord or was handed to them, whether it was an internal improvement project, and whether they had to fight against the establishment to get things done.
So there you go. Values, skillset, competencies, initiative, and ability to execute. You won't necessarily get all of that out of a candidate every time you ask such a question, but there's no such thing as a perfect interviewing technique.
For someone straight out of college or without much experience of course you have to modify how you interview them, but that's always going to be true, and sometimes this same question can still be valuable, just not in exactly the same way.
P.S. Also, just having a good set of questions or a good process to go through is not enough to make someone a good interviewer. Interviewing is hard, easily one of the hardest things in the profession, and only a small subset of folks who do it are actually good at it. I don't know exactly how to make a bad interviewer into a good one, that's a subject that could fill volumes and volumes. But I think this particular kind of question is, on par, much more helpful than not and a step in the right direction.
I don't see why it's a good question. Presume you get an good answer to the question and every followup question. Now tell me this: how much more do you know about how capable the candidate is of doing the job you're hiring for? And, were these questions the fastest, most accurate way of learning that?
I submit: no. "What were the biggest challenges you faced"? How do I assess the answer to that question? How do I compare it with answers from other candidates? How do I track, over time, which challenges correlate to strong performance in the actual role we're hiring for? "Explain your manager's style and whether you liked it"? Come on.
Don't virtually all of these questions ask me to read tea leaves, to get a sense of whether I "like" or "respect" the candidate? That's not the same thing as learning whether the candidate will do a good job.
Oh, and good luck hiring anyone right out of college this way.