What I would suggest is that when the interviewer and candidate get reach the "quiz or coding exercise" part, have them pick a question from a website that provides a question at random, and let both work together towards a solution.
This matches more closely with what they will end up doing anyways if the candidate is hired, and will remove the "I've seen 20 different ways to solve this" bias while also generating empathy for the candidate when the interviewer him/herself also struggles with a fresh problem.
Of course, the interviewer can try to lead the candidate and can hold back from telling the solution right away if s/he can clearly see it, but if not this could set the stage for a fairly realistic way to sample all kinds of qualities, from technical, to communication, to empathy, to teamwork, etc...
I think it would also be less stressful from the candidate's point of view: a candidate often feels like an interview seems unfair because s/he is being asked about something that the interviewer has had a chance to review and prepare much in advance.
When interviewers ask trick/complicated questions, I sometimes wonder what would happen if they were to ask the same question to the rest of their own team. Are they expected to answer it well? Would they know? Think about the places you've worked and the questions you've given at interviews: do you think your own colleagues would have aced them?
Obviously, this doesn't completely remove the additional stress on the candidate, since the interviewer's job isn't on the line, but I think it would provide a more balanced assessment of the candidate's abilities and personality traits.
In practice, I'm not sure interviewers would be very receptive to such method, as it could turn an interview into a stressful event for them.
It all stemmed from one particular interview I had years ago. The interviewer presented me with a very simple problem, I solved it. He then stepped up the difficulty a bit. This kept happening and as it got harder he started acting more like a co-worker where we bounced ideas off of each other.
Granted he knew the solution, but the mere fact that he presented himself not as an interviewer judging my performance but as a co-worker helping to solve a shared problem made that one of my favorite interviews.
They may have changed their interview process at this point though, I only interviewed for them that one time and it was quite a few years ago.
In the past I was hired on the back of simply sitting down with the team lead, and us pair programming to implement what he happened to be working on at the time.
That gave a good idea of how quickly I could start contributing, how easy I was to communicate with, whether I was going to be a snob about the existing code, whether I have relevant pre-existing knowledge about the technologies being used and whether I had any interesting insights to offer. There was also a more standard interview process accompanying this (which I doubt I excelled at) but the pair programming excercise gave a real world insight into exactly what I had to offer and what I was like to work with.
The problem is that getting the answer correctly shouldn't be the main goal of the quiz question. Rather, it is the opportunity to see how the candidate think, and how the person is able to work with someone else (aka the interviewer) to solve a given problem.
It could also be a way to make sure the candidate understands a few concepts of its field (complexity, memory pointers, etc).
Mainly what matters is the process, not the solution.
I always ended those 1:1 sessions by asking if the team member would be comfortable working with people who couldn't get to the correct answer. And if they could, what they would want to see from a candidate working on the question.
Power interviewers do a lot of interviews. They can do them in their sleep and crank them out like an assembly line. But assembly lines aren't nuanced, I fear power interviewers lose the ability (or desire?) to assess candidates through their performance instead of strictly assess the performance.
Power interviewers concern me. I think they end up with too much influence over a company's hiring practices.
I've heard of people at big companies who conduct thousands of interviews a year. This gives them a lot of sway in company hiring practices and culture.
The problem, though, is these large companies are hiring at scale. Growth + attrition yields a lot of hires. Google itself just announced in their earnings call nearly 20k more employees in the past year. That means 100k+ interviews conducted. It's hard to have every interview stay personal an nuanced at that scale.
I don't disagree with the rest of your thesis, but this seems off by an order of magnitude. They would have to conduct 4 interviews every working day to reach even 1000 interviews. Counting the time needed to write feedback for every interview, that person would be a full-time interviewer who occasionally writes software/does product management/project management.
Unless you're speaking of a small group of people who conduct disproportionately more interviews than everyone else (Pareto distribution) - a dozen interviewers could easily rack up 1000 interviews between them.
I admit it's hearsay, but yes, I've heard that some people supposedly conduct 1,000+ interviews.
To be fair, I'm sure some of those are the online "solve this coding puzzle in 30 minutes" type that can be watched later at 3x speed and don't require human interaction. I don't know the ratio, maybe the power interviewers are heavily sandbagging with those.
This is an interesting point, because I feel like that's how large companies treat employee performance in general. I often hear stories of talented engineers being treated as cogs or passed over for promotions, such as the classic protobuf maintainer example.
Perhaps the assembly line fashion of interviews is reflective of how large bureaucracies treat employees as a whole.
I would be sad if I had to switch questions after 20 or 30 candidates because I find it necessary to invest substantial effort in calibrating a new question. Before I ask a candidate a new question, I try use it to mock interview at least 10 people I have worked closely with. Iteration is always required to tune the difficulty and complexity of a question, so the total time invested in a question can be quite high.
One way that I keep questions well-calibrated is that I use them to mock interview other members of the interview panel. This serves at least two purposes: one is to constantly remind myself what realistic answers sound like and another is the help the rest of the panel understand the areas my question will cover.
I find that — for me — this type of mock interviewing and the subsequent retrospective cultivate empathy for candidates. I think this avoids the sort of bias you're observing.
(NB: Some of this may be specific to the kinds of questions I ask; I care less about the initial answer a candidate gives than their ability to self-assess their answer or incorporate feedback to improve their solution.)
If you're interviewing correctly, your job as an interviewer is to ensure that the candidate succeeds during the interview. For example, one question might require to check if two intervals overlap. There are multiple ways to check if they do. One can enumerate all possible ways that they can do (first interval fully contained within the second one, second interval partially overlapping with the first one, etc.) but this quickly grows into a complicated conditional. At that point, if the candidate struggles, you can mention "how would you check that two intervals don't overlap at all?" which is a much easier test that can then be inverted by the candidate.
What the interviewers should be looking for is if the candidate can think through a problem, can split it in smaller steps, can solve each step and is able to integrate each small step into a complete whole.
You could do that some process, with the interviewee.
It would be a great way to see how collaborative they are.
i.e. you say "I've never actually worked through this problem myself, so I don't know how deep the rabbit hole is, but let's give it a crack and see what we come up with together."
Sounds a lot more like real-life to me than a manufactured question you've already worked through to the nth degree.
Without establishing objective criteria with which to evaluate candidates, it's much easier to fall back on decision making processes that are prone to unconscious bias.
A manufactured question may seem impersonal, but that's the point.
Sometimes, all the contestants will miss a prompt, and Alex Trebek comes across as a bit smug or incredulous when reading out the correct response. ("No one? No one knew this? The correct response is....") I may have occasionally yelled at my television, "You have been the host for decades! They have to give you all the answers in order to run the show!"
And that's what it is when you think you're smart just because you know most of the answers on "Jeopardy!" The game is different when you're watching it (or hosting it) than when you're actually playing it competitively.
The interview processes that are designed to stress or fluster a candidate can make the divide between host and players even worse. There is nothing the candidate can do to get the interviewer fired. Almost anything the interviewer does can result in a no-offer.
https://www.youtube.com/watch?v=wwIysnVmAUg
Then go interview with them. They don't stand next and engage you while you're working at the white board. Instead, they sit at the opposite end of the table and type out your code line by line on their laptops so at the end they can see if whatever you scribbled actually compiles.
The laptop requirement just encourages disengagement. I knew the problem and how to get to the optimal solution for my first round at Google so it didn't affect my performance, but I was pretty disappointed because my interviewer, between typing up the lines I wrote on the board, was completely occupied answering email and Slack messages and barely said anything or made eye contact aside from the occasional "mhm" or nod. Had I gotten stuck at any point I don't think I would have been able to get any help, certainly not any help like I would get from a coworker in a truly collaborative scenario.
$300k+ total comps are a big part of it
I would take a deep breath and go at it, thankful that this unfriendly person isn't breathing down my neck while I'm solving the problem.
Although that hardly constitutes a test of how someone will work in a collaborative environment.
Real collaborations have a lot of tangents, and rabbit trails, and clarification requests, and elucidation sidebars, and restating what has already been established, and re-asking the core questions to see if they have been answered. If the other person is just nodding and saying "Mmhmm. Go on.", then that's no "true [Scotsman]" collaborator. It's a typical collaboration, though; there are two people in the room, and between the two of them, someone is doing the work.
My experience was that the video might show the ideal interview but not the default one. The default was tense silence and "okays" that barely restrained their judgement. The linked video seems to give an entertainingly wrong impression.
Or perhaps you got to find out exactly what it's like to work with that particular person.
I think one fix is to have an explicit criteria scoring a given question, and training interviewers to not rely too heavily on a single question.
I noticed I had this same issue, that I would give interviewees a pass if it was a sub-field I was unfamiliar with(javascript at the time) and more critical if it something I was really into.(leveraging the type system to prevent different classes of errors).
This is so incredibly unprofessional I'm shocked that this large search and ad company wouldn't pull this interviewer out of the rotation.
I hope this was just a joke. If I received that in feedback for a candidate I'd replace that interviewer in a heartbeat.
The fact that the organisation not only has people who are like this in a professional setting working there but also tolerates it doesn't reflect very well on the organisation. I understand it's difficult to control for the variation in personalities working at a large company but it's not too difficult to set a culture where a base level of professionalism is expected and is the norm. I'm not sure of how reviews work at your company but I sincerely hope comments like this officially flag the person as not suitable for management and possibly not suitable to continue in long term service at the company.
I could tell you which questions were asked at MTV this past week and by looking at the last month of these posts for a specific office it's incredibly easy to come up with the most frequently asked questions and memorize their optimal solutions. This means that if you're someone who doesn't partake in these communities that you'll enter these interviews at a significant disadvantage because your performance on that problem will be compared to everyone who was able to (legitimately or illegitimately) arrive at the optimal solution in 45 minutes.
Even the most common questions aren't aren't common enough that you're sure to encounter one on an interview loop.
1) At the office that my friend (who is also an interviewer) works, there is a pool that the interviewers collaboratively built and generally select from. They aren't required to choose from this pool, but 95% of them do because they contributed to it.
2) At MTV there's either a pool or the number of interviewers is so low that the same questions are repeated over and over at high frequency for several weeks, making it trivial to know which ones are likely to show up. I know because the questions I found on the sketchy Chinese forums were exactly the ones I encountered on my onsite interview.
I'm unclear if 1 is referring to a specific office pool, which would be weird, or the global knowledge base I mentioned, but that has lots of questions, more than one could hope to memorize in any reasonable time frame.
I would like to throw an experiment your way. As you mentioned, the interviewer, and interview questions itself can have its own bias. How about pre-screening questions? First, you would pool the possible set of interview questions. Then, once a month, a meaningful subset (or subsets) of 'regular' interviewers is created. Each subset gets one question from the pool and are required to complete it in 'interview like conditions' (closed room, no internet, just pen and paper). If the question has a poor answer rate with existing employees (aka interviewers), throw it out. It would be great if they explained what they did and did not like about the question and what would have made it easier. Would this process increase their empathy towards the interviewee? It needs to be frequent so it is a constant reminder.
Another, likely chaotic, option is to have the interviewer be randomly assigned a question from the pool. The question cannot have been their own submission. This way, both the interviewer and interviewee are seeing it for the first time and have to work toward a goal (team work).
Changing topics from interviewing to interview metrics... I would be interested in how interviewee - interviewer age differences affected outcomes.
I've ignored every potential job ad I have seen from said search and ads company ever since.
Would it also make sense to have every harsh interviewer be interviewed by all the other harsh interviewers, and see how well they all perform on each others' question? In each case, incentivize the interviewer to stump the interviewee, but using only the same questions they use in real interviews with prospective candidates, and link the interviewee's performance to their bonus or something. Put the harsh interviewers in the same situation with a similar amount at stake as prospective candidates, remind them that nervousness and the unnatural setup relatively to typical working conditions can also be a factor in interview performance.
I find it curious that you think this has to do with the particular question. This kind of bias comes up all the time, and it will certainly be in place even the first time you ask a particular question.
Simply growing in one's career and being around mostly other experienced people can quickly lead to one forgetting what it was like to be new to the field.
It's total lack of empathy at that point, and if the candidate doesn't exude near-perfect interviewing brilliance on that specific question, the interviewer judges them as essentially worthless.
That's going way too far. Having poorly calibrated standards doesn't imply lack of empathy. Having an unreasonably high bar doesn't mean you think anyone under the bar is worthless.
There are some other experiments I'd like to run. One of them is to have interviewers go through a one-hour interview themselves for every 50 or so interviews they give. Maybe match up an interviewer who has a track record of being especially harsh on candidates for not giving a flawless performance on the question they've been asking for a while. The idea is to see if we can't bubble up some empathy.
How about just recording data about interviews and using it as feedback to make the system better? If there are outlier interviewers with unreasonable interview-to-hire rates, deal with that situation directly.
The question is not "is this person good?" or "can they do the work?" The mandate is not "hire people who can do the work” and rejection is not "we don't think you can do the work.” The mandate is "hire people we're confident that we're excited about, even with the imperfect information we can afford." Like a competitive college, we're going to pass on a lot of applicants who are probably fine. It's not a reflection on their worth as people or as professionals. It's a pragmatic tradeoff in the design of a machine.
It behooves all parties to cast their nets far and wide, and to not get emotionally invested in any particular match before the offer stage.
My first dev job was at a place that had a whole team like this. They would get themselves psyched up for interviews and code reviews talking about making someone cry today.
It was pretty disturbing.
I'm glad most devs aren't like this, but sadly, there are still too many who are.
For example, let's say a previous candidate that was brought in as an L4, rose to L6 extremely quickly while constantly getting great performance reviews - maybe you could look back at the type of questions asked, and the type of feedback that candidate received.
Additionally, let's say a previous candidate ended up being a non-performer and was quickly let go, you could also look back at their initial interview.
Maybe everything could be fed into ML, and once you have a model in place you can start getting signals based off a candidate's replies to behavioral interview questions, or certain characteristics displayed during their systems design interview, etc.
- Review candidate's resume and side projects to get to know more about them. This helped with getting to know their work and finding things to discuss beyond the interview exercise.
- Before diving into the exercise I would out right tell the candidate what I was looking for in regards to the exercise; e.g. "I'm not looking for a complete/perfect answer but I'm looking to have a conversation about the pros/cons and edge-cases. You can write on the whiteboard as little or as much as you want to but either way let's have a discussion. Let me know if you feel stuck at any point, and I'll make sure to let you know if you are/aren't on the right track. If you don't know/remember something just ask me; it' fine, nobody knows everything."
- I didn't put too much weight on whether the candidate gave a complete answer or not, or how much I had to help them. I basically asked myself a simple question: "Do I feel like this person would be a productive contributor here and are they someone I would be able to work with?"
- I always did my best to go into the room with a relaxed and conversational attitude. I was there to pass the candidate and not to fail them for random reasons.
- I passed most people and only failed some when it was somewhat obvious to me that they really lacked some very basic foundational pieces, or when I felt like they weren't someone I would want to work with (for various reasons).
Towards the end of my career there I started to really dislike interviewing because I would personally put so much effort to passing candidates, but they wouldn't get hired because other interviewers left feedback like "I helped the candidate too much" or "they didn't even know what a TRIE tree was" or "they struggled with X" or "the solution had bugs" or "the solution wasn't complete".
They interesting thing was that the same interviewers that failed candidates for seemingly random reasons, were also the ones who left confusing feedback or not enough feedback. Also the same interviewers either treated candidates poorly or they showed boredom and agitation.
Care to share the name? The search engine I currently use is garbage, it routinely ignores half my query.
This would only be effective if the interviewee faces similar consequences for failing that interview, e.g. losing their job (the equivalent of an actual interviewer not getting an offer), or at the very least losing the ability to interview moving forward.
They might also fix their comically bureaucratic promotion committee process at the same time. Seems like a win-win.
Maybe it’s time to step back and admit that Google-style interviewing is itself intrinsically toxic and unhuman? It brings out the worst in everyone, reduces candidates to rote memorization and tortured puzzle solving, and very sincerely does not reflect the realities of the job (not even at Google).
> There are some other experiments I'd like to run... to see if we can't bubble up some empathy.
I don't know dude, it doesn't sound like you're the authority on empathy if you think you can "bubble up some empathy."
As far as making fun of the preposterous egotism of the Google interviewing process is concerned, that line could definitely make it to Silicon Valley in the parody.
I mean, I don't really know. If you aspire to be better at recruiting--of being a people person of some kind--you gotta not say stuff that sounds utterly, hilariously disconnected from what normal human beings say.
In my position, I get the occasion to read a ton of assessments written by interviewers. Some of the most striking assessments are the ones where the interviewer is cock-sure that they completely nailed the candidate's utter and complete incompetence. It's usually an interviewer who's been asking the same question dozens of times over a year or more. They've seen every variation of performance on the question, and they've completely forgotten what it was like for them when they first encountered a question like that.
It's total lack of empathy at that point, and if the candidate doesn't exude near-perfect interviewing brilliance on that specific question, the interviewer judges them as essentially worthless. Interviewers like that sometimes even get snarky and rather unprofessional in their writeup, "Finally, time ran out, mercifully ending both my and the candidate's misery."
If I were to diagnose one of the causes of this phenomenon, I'd say it is bias. The interviewer best remembers the candidates who performed exceptionally well on their question, triggering the availability heuristic.
There are tactics that I think can be effective to bust those biases. One might be to put an upper limit on the number of times an interviewer is allowed to ask any given question. Once they've asked maybe 20 or 30 candidates the same question, it's spent. They have to move on to something substantially different.
There are some other experiments I'd like to run. One of them is to have interviewers go through a one-hour interview themselves for every 50 or so interviews they give. Maybe match up an interviewer who has a track record of being especially harsh on candidates for not giving a flawless performance on the question they've been asking for a while. The idea is to see if we can't bubble up some empathy.