Phenomenally good advice. I wonder, and I don't mean this to be a loaded question because I really don't know the answer: do other practitioners in other highly-skilled disciplines have to dance these sorts of dances to get hired? Does a doctor have to diagnose on the spot? Does an architect have to design a building on a whiteboard? Do they make a lawyer stand up in front of a group of partners and argue some case off the top of his head? Perhaps they do. I really don't know. But it seems to me that basically professional experience counts for little other than to keep the resume out of the trash folder.
Anyone can be a doctor or lawyer too. I know what you mean, I think, but I don't know if I buy the distinction. There is a formal pass-the-bar testing event for lawyers. For nurses and I believe doctors there are boards. Same basic purpose AFAIK. They take these tests when they are just entering the profession. How meaningful are they years later, compared with the experience gained in the interim? My wife is a registered nurse working in cardiac critical care for seven years. If she interviews for a new job with that employment record behind her they aren't going to ask her what an aorta is, which is sort of the equivalent of what Asana is talking about doing here (and what everyone does, to be fair).
So you seem to be saying that a professional nurse, as an example, can be trusted to know the basics after seven years of work, because he or she was required to take a hard test on graduation. But a programmer with seven years of experience must prove that he or she knows what a binary tree does? It's also worth pointing out that lawyers can get people jailed or cause them to lose their property. Doctors and nurses can kill them. So can we, occasionally, but it's rare.
The most effective interviewing technique, as far as I am concerned is to send the candidate out for lunch with a few of your engineers and tell them to find out whether he knows what he's doing. You spend an hour talking software with some people and it's awful hard to fake knowing your stuff.
you'd be surprised at how many programmers with seven years experience have no freakin clue what a binary tree does!
There is a reason fizzbuzz is phenomenal filter.
fizzbuzz does not filter for programmers capable of doing original, thoughtful work in a realistic work environment.
A developer (in a non-pathological work environment) has time to discuss and research a solution.
FWIW I don't remember the last time I had to implement a binary tree since my C++ course in college; I'm not a fan of pop quizzes in interviews, but if you're going to do them then at least make it relevant to the work you do on a daily basis.
It's like leaving money on the table. Someone needs to moneyball the crap out of this.
I agree! (I don't even think it's counter-intuitive at all.) Thanks so much for really clear feedback.
Here's how.
Some problems - particularly those involving concurrency - don't lend themselves well to the standard compile/test/run/printf mode of programming. The more difficult problems you'll run into require some semiformal reasoning about the underlying problem. When I sit down to solve these problems I'm doing more or less exactly what Asana is asking for. I'm not making my code compile. I'm not running tests - with high probability they'll pass even if my code is wrong.
Eventually I'll need to actually write code which compiles and runs. But if I skip the "think hard and make sure it's right" step (what Asana is asking for) I'll wind up with wrong code in production with errors that are nearly impossible to debug.
It sounds like Asana wants to determine if people can do this kind of thing.
Same for Facebook/Google/Apple/Microsoft/Amazon/etc that also use whiteboard interviews. Those types of companies value seeing candidates' thought process more than the final code of work-at-home test questions. There's a signal there in whiteboard interviews that can't be replicated by at-home tests. Whether that unique signal is valuable or worth the cost is up for debate (as the ongoing disagreements about hiring practices show).
What I dispute is that anyone really does that.
It is awfully easy to rationalize any hazing ritual with this sort of six- degrees- of- concurrency- and- formal- methods logic. But there's a simple observation to make about all these questions: they do not mimic the experience of doing the work the interviewer is evaluating candidates for.
The more a question diverges from the experience of doing the job, the more process and framework and structure and rigor is needed to extract value from it. But really: most of these questions are just the same dumb fucking programming questions, just with different variants of the interviewer repeatedly whacking the candidate over the head with a paper towel roll while trying to answer.
I've interviewed at startups that have some of the best work-sample testing that I've ever seen - but I could immediately tell these systems wouldn't work well at larger companies.
There are entire companies devoted to helping people prepare for GooAmaBookSoft interviews. If any one of these companies standardized on a single interview for all candidates it would immediately turn into a situation like standardized testing in high school where students study to the test instead of learning the core material.
I guess my advice is that if you're a startup, take advantage of this! You can do things that don't scale. Interview in ways larger companies can't. Don't assume that the way these companies interview is the best for you.
As another example, I ask people to set up numerical/statistical algos without worrying about specific numpy details. I trust people to translate a log likelihood formula to python, but I want to see them actually derive that formula. This isn't some hazing ritual - it's important to actually understand the statistics to avoid getting stuff wrong.
Again, the tools won't help you - if you set things up wrong, you just get a bunch of false positives.
I understand this isn't very important for the average dev writing CRUD or ETL apps. I'm just pointing out a particular use case where their constraints are reasonable.
So if deriving log likelihood formula in high pressure, time limited situations, is a good proxy for the kind of work the candidate will be doing, by all means do it that way.
If on the other hand, they are more likely to be asked on the job, to go away for some time, think about the questions needing answers in a data set, design an algorithm to suss out those answers and then persuade their colleagues that it is correct, you should design your interview process to align with that.
We’ll ask you to solve some coding questions in a language and text editor of your choice. Feel free to bring your own laptop, or we’ll be happy to provide one. Notably, we ask candidates not to compile or run their code during this exercise, and not to refer to online resources. Our goal is not to simulate day-to-day software development — where we read docs and write lots of tests! — but rather to see how you reason about your code and input cases. For that same reason, we won’t ding you for superficial syntax errors or misremembered function names. After leaving you to work through the questions on your own, we’ll sit down together and talk through your solutions (including any ideas you didn’t have time to commit to code).
In any case, all I'm disputing is the idea that formal reasoning about algorithms (without stressing function names and compilation) is a useless skill. We have tools that make it unnecessary for many entry level devs, but that doesn't mean everyone can ignore it.
To me it sounds exactly like they want you to demonstrate some semi-formal reasoning about algorithms, and you think this is not a realistic representation of work. What do you think they mean, and what do you oppose?
Specifically:
It's clear from the tone that you're trying to mitigate the hostility and unfriendliness of the conventional programmer interview. Great! But you need to do more than superficially adjust your tone. What this document describes is literally the circa-1999 software developer interview, in almost every detail --- the fuzzily described coding question, the "algorithm and data structures" interview, the design review.
Everyone has heard of interviews that pointlessly refuse candidates the opportunity to consult Internet references, like every working programmer does every time they code anything ever. But your process goes a step further: you won't even allow candidates to compile their code. How can this possibly be helpful to your process?
The most alienating and hostile aspect of the conventional job interview is on-the-spot whiteboard programming. You shouldn't have programmers write code in interviews at all, if you can avoid it; instead, take a week or two and design programming questions that candidates can do at home, on their own schedule, in their own surroundings.
But if you're going to make people code in an environment that is utterly unlike the one professionals actually work in, the onus is on you to do everything you possibly can to mitigate the performance penalty you're imposing. Why not, instead of coming up with crazy rules like "no running your own code allowed", point your creativity in the opposite direction and find ways to get candidates to be more at ease with writing code with someone looking over their shoulder?
Here are some things you can do right now to make your interviewing guide actually helpful to candidates:
* Provide sample questions. They do not need to be the ones you're asking in real interviews (although: if they are truly good questions, they could be!), but they should be close enough that a candidate would have no business being surprised by the real ones.
* Provide a detailed breakdown of your interviewing process. Do you phone screen? How many times? Who staffs the phone screens? What kinds of questions get asked on them? Who delivers the in-person interviews? How long do they last? These are some of the questions real candidates have about interview processes. I don't understand why more companies don't just answer them up front.
* Provide reference material for candidates. Don't point them to other people's interviewing guides! You know your jobs better than anyone else does... right? How about instead: what are the most popular books on the shelves at your team? (Bonus points: buy those books for candidates). What are some Github repositories you think represent really well-engineered software? What are some examples of really hard problems your team is still grappling with?
I hired for years and years with a battery of over-the-top technical questions and a rolodex of personal contacts. Then my team and I had a crazy idea: we opened our whole interview process up, standardized it, and took pains to make it understandable to people with no experience in our field. What we learned was that there is a huge amount of underutilized talent out there that can't (because it's locked out due to jargon and poorly described requirements) or won't (because it's denied social permission) apply for jobs unless you go out of your way to bring them in. Many of these people are better than you are. At least, they were for me. Stop bloodying your forehead against the wall competing with Facebook and Google for people who --- if they can navigate the process you describe --- can get an offer from those companies any time they want.