My favorite interviews were when we were trying to hire a senior level engineer, and we were only interviewing people with 10+ years of development experience. We'd ask people to build a priority queue. Items with a higher priority need to be handled before those with a lower priority, but otherwise items should be handled in order.
Can you guess what the most common methodology people chose was?
Did you guess an XML document?
These are people with degrees in comp sci, some claiming 15 years of experience as a developer, trying to get a role at a top 10 bank in New York City.
Of course technical interviews shouldn't be the only test used for hiring, but they absolutely should be included. Maybe it's demeaning that you have to jump through these hoops. Or maybe you're bad at them. Or they're "beneath you" in some sort of way. But don't blame the interviewer, blame the hundreds of other people trying to get a job that they aren't qualified for.
A: We need programming quizzes even for senior engineers, because many turn out of be incompetent.
B: How do you know they're incompetent?
A: They failed our programming quizzes!
The reality is that you're testing for stage fright, not programming skills. An audition is not a good criterion for choosing people who aren't stage performers.
I'm betting it was something like this though: https://www.chegg.com/homework-help/questions-and-answers/pr...
Assume good faith: let’s presume he knows some candidates get stage fright and has ways to help mitigate it. Or that despite stage fright, such tests are still a good signal (e.g. the signal is so strong the false negatives are a worthwhile cost, or stage fright might be highly correlated with poor performance, or stage performance could be highly correlated with talent!)
That's an interesting defense of an interview process that specifically does not assume good faith of the candidates.
> has ways to help mitigate it
I'm not sure this understands the nature of true stage fright. There aren't any good ways to mitigate it.
Re mitigation: already answered with previous “or” clause.
It's like if someone has a fear of spiders, and you bring out spiders during an interview but claim "we have ways of mitigating this fear."
I think that the idea is to optimise for preventing false positives (bad hires), at the cost of a higher rate of false negatives (good hires incorrectly weeded out). Interviews are imperfect: candidates can fail for a vast range of reasons that are completely unfair.
Also an applicant could train to avoid stage fright, just as they might train to learn algorithms if applying to Google. AFAIK many real actors have all the symptoms of stage fright, but they have learned to deal with it to be successful.
Disclaimer: I am not an interviewer.
Note that this thread started with "Everyone loves to read and write about how developer interviewing is flawed, but no one wants to go out on a limb and make suggestions about how to improve it." Now you're saying applicants can train for this, applicants can train for that, but what happened to the original point about how to improve the process instead of making candidates take difficult steps to adapt to the flawed process?
Anyone who thinks XML is a valid solution to a priority queue is incompetent (even if they were prompted by the recruiter as said in sibling comment). https://news.ycombinator.com/item?id=24758234
Stage fright is real, and it is unavoidable in an interview i.e. even if coding quizzes were removed then the problem still exists, so arguing just against quizzes is silly IMHO.
> everyone claims are acceptable casualties of the system
I would say instead that casualties are unavoidable - interviews must be imperfect because time is limited (regardless of how much we may try to improve interviews.)
> systematically discriminate against people who tend to experience extreme nervousness during auditions but otherwise perform normally on the job.
Interviews are by definition systematically discriminatory: one is trying to select one person. The data is dirty, and if you make interviews more inclusive on one dimension you are discriminating against others on that dimension. They must be systematic as per anti-discrimination regulations, or else you will get sued to your cost.
How to improve interviews to reduce false negatives is hard, but I think you are oversimplifying it down to “quizzes are bad”, which is at odds with many smart professionals who say they are good for detecting incompetent developers. You have not suggested any improvement that detects incompetence.
> Note that this thread started with "Everyone loves to read and write about how developer interviewing is flawed, but no one wants to go out on a limb and make suggestions about how to improve it."
I can also cherry pick a quote further up-thread: “To many people, the coding interviews is a chance to break the traditional mold and land a job that would have been out of reach otherwise.”
There's only so much that can be said in a HN comment. I'm trying to avoid writing a whole treatise.
> Stage fright is real, and it is unavoidable in an interview i.e. even if coding quizzes were removed then the problem still exists, so arguing just against quizzes is silly IMHO.
The conclusion doesn't follow. The problem still exists, yes, but I think everyone would acknowledge that casual chatting is easier than coding quizzes? Thus, if you tend to experience performance anxiety, then tasks that are more difficult to perform are more strongly negatively affected by performance anxiety. Compare with other issues that affect performance, e.g., drunkenness. Being drunk will affect your speech, but it severely affects your ability to more complex tasks such as driving a car.
> Interviews are by definition systematically discriminatory: one is trying to select one person.
That's not what systematic discrimination means. It means discriminating against a certain class of people who all share the same characteristic. Some examples are gender, race, and age, but that's not an exhaustive list.
> You have not suggested any improvement that detects incompetence.
I fundamentally disagree that this should be the main purpose and focus of hiring and interviews. I think rampant, exaggerated paranoia is one of the reasons that tech hiring is so messed up. The irrational fear of the "bad hire". Few other industries are so afraid of this. The reality is that it's impossible to avoid bad hires, and trying to be perfect in this respect will distort the whole process. Professional sports spend vastly more time and money on talent evaluation and acquisition than tech, but still they make "bad hires" all the time. Wasted #1 draft picks. Multi-million-dollar contracts for busts. It's just expected, a cost of doing business. They shrug it off and move on, and keep making mistakes too, because perfection is unattainable.
"How can we drop coding tests... while still being ultra-paranoid?" is not a question I can answer. My answer is to not be so paranoid, and just accept the inevitable mistakes, a more relaxed process which also allows you to find the "diamonds in the rough".
> I can also cherry pick a quote further up-thread: “To many people, the coding interviews is a chance to break the traditional mold and land a job that would have been out of reach otherwise.”
I don't know why it's cherry picking to restate the original context into which I first replied in this thread. In any case, I already replied to that quote too.
I am going to throw in a patio11 spanner instead: ‘In a way, every scaling startup is an experiment in empirical microeconomics research on “What parts of the typical corporate form are necessary and which are pageantry which we only keep around due to anchoring, the sunk cost fallacy, and tradition?” Every time a startup bites the bullet and hires a VP of Sales, a lifecycle email copywriter, a retirements benefits administrator, or a cook, count that as a published result saying “Yep, we found this to be necessary.”’. https://www.kalzumeus.com/2020/10/09/four-years-at-stripe/
If you aren't fluent at coding then why are you applying to a coding job? Would you apply to an English writing job without being fluent in English? If you just have a few years of English from school doing an interview in English would be extremely stressful and likely lead to stage fright. That doesn't mean that making interviews in English is a problem, the problem is that the person isn't fluent in English.
I remember that English fluency was a big barrier when I originally applied for jobs. Just speaking took a lot of mental power and the interviews were very stressful, but now that I'm fluent it no longer takes any effort. I went through the same stages in programming as well. I did a technical interview with a few months of coding experience and it was very stressful, now I don't even have to think.
I'm just going to link again to a scientific study on the subject.
Edit: Also looking at the data people spent more time on the problem in the private setting than the public. Their stats could therefore be as simple as the students not caring about the result and therefore they quit earlier in the more uncomfortable environment instead of doing it properly and checking their solution. This wouldn't be a thing in real interviews since there a lot is at stake so people wouldn't quit early, at least in my experience not a single candidate I've interviewed quit early.
Edit Edit: For example, their very impressive "All women in private completed it, all women in public failed" stat can also be described as "Women in private spent on average 16 minutes on the problem, women in public spent on average 7 minutes on the problem". Maybe if they had more at stake and therefore more stress they would have spent the entire 30 minutes and therefore done just as well?
Anyway, my point is that just because the abstract says they have clear evidence that stress is hurting performance these studies are still mostly garbage. Don't take it as proof just because there is a study.
I don't. I take my lived experience as proof that this is real.
You seem to be in denial of a common psychological experience.
Exactly!
If someone has been implementing and delivering products for years/decades and they "fail" a whiteboard puzzle, which is more likely?
A: The whiteboard puzzle doesn't actually measure anything correlated to delivering products.
B: They can't actually develop anything, despite having repeatedly done so.
I want to hire the people who can develop and deliver products, not the whiteboard puzzle specialists because I've never been at a company where solving whiteboard puzzles is something we do as part of the business.
There may be a few small niches that attract good quality talent, but otherwise I don't know why anyone would opt to work as a technologist at a bank as opposed to say, a bonafide real tech company (all the banks are eager to say they are "tech companies" nowadays), or even a hedge fund.
> Did you guess an XML document?
If that is your most common experience with asking that question, maybe it is time you consider whether you are communicating what you want properly to your candidates.
But I don't know how much better I could have described what I wanted:
"I want a class for holding a queue of items in memory. The queue will only need to hold a few thousand items, and you don't need to worry about persisting it in case of a power failure or anything. The only wrinkle is that when an item is added to a queue it has a priority. Higher priority items need to come out of the queue before lower priority items, but items with the same priority should come out in the order they were inserted."
My naive contribution: A linked list with pointers to the tails of various priorities. The grabber just points to the head and pops things off the top as it goes.
I tried. It got tiring. Sometimes people would loop through all possible priorities plugging them into an XPath query and pop the first element of the highest priority that matched. Some would do two XPath queries, one across all nodes to get the max priority and a second to query the first element (ignoring the lack of a need to run the query a second time at all). Some would insist that XPath has some function that would let you select a node with an attribute that has the maximum numeric value.
I'd have been fine with your solution.
Every job gets applicants that aren't qualified. Only software engineering out of every single non-performance based (musicians, actors etc...) industry out there has white board interviews.
It was an unmitigated disaster. Constantly had to double check his work and redirect him from doing things bad ways (like using mutable objects with changing hash codes as dictionary keys kind of bad).
Eventually I quit the job (for other reasons). On the last day he cornered me in the elevator and offered me $100/hr to do his work for him under the table so that he could claim it was his.
> Did you ever verify that they actually had the experience they claimed?
Employer would only verify employment dates and eligibility for re-hire.
> Did you follow up with their references?
Yeah, but when you ask someone for references they're not likely to give you people who wouldn't speak well of them.
> Were they able to walk you through projects they worked, or discuss solutions implemented?
Yeah. I can walk you through projects that I was involved in as a junior as if I had been the lead because I paid attention during meetings.
How many bad experiences have you had with someone who did? I've had enough that I've lost count.
>like using mutable objects with changing hash codes as dictionary keys kind of bad
I don't think a leet code question would necessarily have screened stuff like that out at all.
>Yeah, but when you ask someone for references they're not likely to give you people who wouldn't speak well of them.
Did you actually talk with these references about what it was like to work with him?
>On the last day he cornered me in the elevator and offered me $100/hr to do his work for him under the table so that he could claim it was his.
That's $200k a year. I find it pretty hard to believe that he was serious.
It was a bad experience with someone who utterly failed a whiteboard.
> How many bad experiences have you had with someone who did? I've had enough that I've lost count.
> I don't think a leet code question would necessarily have screened stuff like that out at all.
I'm sure I have too, but I never advocated for only whiteboard interviewing. There should be a spread covering all sorts of aspects of their abilities.
> Did you actually talk with these references about what it was like to work with him?
Of course. I have a wide variety of coworkers with a wide variety of opinions of my skill-set. I'd select the ones that I think would resonate best with you. Barring that I have half a dozen friends that would tell you whatever I want them to. I fail to see how references are any sort of magic bullet.
> That's $200k a year. I find it pretty hard to believe that he was serious.
I was another senior engineer on the project, so I definitely could have covered the work. I had deeper experience, and given his quality I could have churned things out more quickly. I suspect he wasn't offering me $100 / hour at 40 hours per week. The job had a lot of paperwork and bureaucracy involved. I suspect if I was just doing the coding parts I could have covered it at about 10 hours per week, which is a lot more feasible.
Obviously I turned this down for ethical reasons.
Could it be that you have people who will give you good references because you're a good developer?
>Barring that I have half a dozen friends that would tell you whatever I want them to.
So if I pull those friends up on linked in, they're developers too, and you'll have some plausible overlapping work history?
> I fail to see how references are any sort of magic bullet.
It's not a magic bullet. But faking 10 years of development work history and multiple decent quality references is a fairly high bar. It's not going to catch everyone but neither is whiteboarding. The goal shouldn't be to catch everyone.
Sometimes you make a bad hire. Despite every test you put someone through, you'll get false positives. The problem is that if you ratchet your test up to far to avoid false positives, you end up turning off good engineers, or failing good engineers.
I think the problem is that the company wasn't willing to just admit they made a mistake and fire the guy, not that he was hired in the first place. America (assuming you're in the US) has nearly unrestricted at will employment. Why are you so afraid of hiring one bad employee? If you can't identify bad employees after a few weeks of working with, you have zero chance identifying them before you hire them.
Maybe instead of spending months trying to find the perfect candidate, just do what literally every other industry does and hire experienced candidates based on reputation, resume, talking to them about the job and past projects, and fire them if it turns out they were completely lying to you.
I remember how to build one from a binary heap, but I wouldn't launch right into the code without reading a bit first.
If I was doing it today I'd use some sort of balanced tree using the priority to navigate the nodes. Each node would have pointers to the head and tail of a singly linked list. Insert when the priority isn't in the tree is a simple tree insert with head and tail pointing to a new single-element linked list, insert when the priority is in there is adding it to the tail of the right node and moving the tail pointer, removing is finding the right-most node, taking the head and advancing the head pointer by one element. If the list is now empty then delete the node.
But none of it matters. I don't care if someone would build it the way I would. I would never expect that. I don't even want an optimal or semi-optimal answer. I'd have accepted a singly linked list with O(n) insertion time and O(1) pop time. Or a plain list where new items are added to the end and a pop is an O(n) seek through the list.
What I care about is whether the developer come up with any solution, describe the tradeoffs that they are making, why they are making them, and what research they would do if any to find a better solution. If i could have this conversation that we're having right here, right now with them then they'd probably pass my hiring bar.
That's why I said it's likely the person who built it would done a bit of research. When hiring someone I care that they remember enough to realize that and are capable of doing the research, more than that they remember exactly how to implement some data structure.
>If I could have this conversation that we're having right here, right now with them then they'd probably pass my hiring bar.
I think that's a totally reasonable. My problem is that this isn't what people are complaining about when complaining about white board interviews.
99% of the time the interviewer is asking for the answer to a very specific problem that is highly biased towards new grads who would have just seen it on a test.
And even if they say you don't care about the optimal solution they will always hire the person who gets the optimal solution in 15 minutes b/c they memorized it versus the person takes 30 min to bang out something that kind of works.
Here's my approach:
I ask someone else on my team to come up with an interview problem, so I haven't seen it before hand. Then I work with the candidate on hashing out a solution.
Hiring is meant to have fidelity. Fidelity to what? To the extent there are company values, nobody explicitly “tests” for them in hiring. Certainly when I worked at US Engineering, which had an explicit values statement, I don’t recall being interviewed for how I “exhibit integrity in everything we do” or “ensure a safe working environment.” I doubt Amazon is like “so let’s talk customer obsession. when is the last time that you stalked a customer and got obsessed with making their life easy?” (I mean you can question the value as well, but to the extent that Amazon tells you its core values, are they a part of hiring?)
How much fidelity are we talking? Use the “nines” metaphor. Do you want five nines, one bad hire in a hundred thousand? Probably you need a month-long “interview” process then, you’re basically at the question of “let us contract some work out to you for a month and if we like you we will hire you.” But, consider: are more nines even better? If we have BS metrics that do not properly reflect our interests then we only find out about them by hiring people we would not have rationally hired. Maybe you only want one nine, or more radically even half a nine (that is 68%, if you have never heard of it), of fidelity. What if you baked that fidelity in? What if a unilateral consensus to not hire still generated a 10% chance of hiring and a unilateral consensus to hire still generated only a 90% chance of hiring, so that interviewers had to know getting into this that absolute sureness was not a goal?
That brings up an even better question, which is: can you have a good hiring process without a good firing process? How do you give yourself the one-nine process and then make sure that people who were retroactively not a good fit get a decent severance but are done away with?
I have mentioned elsewhere that Bayes’s theorem suggests that evidence should be measured more or less in decibels; this is covered explicitly in Jaynes’ Probability Theory ch. 4. But consider [1] where researchers found that they did not see substantially more correlation between different interviewers evaluating the same candidate for 20 minutes, and other interviewers watching videos of those interviews over only the first 20 seconds (the initial greetings of the video). So can we actually trust interviewers to give us, say, +10dB of evidence that this person is a good fit? Or if we get 5 interviewers do we get a range from “meh, pass” to “hire immediately for twice what they’re asking” as much as if we had done job interview “speed dating”?
Constructive advice is hard. One wants to instead force the hiring committee to watch talk after talk by Alan Kay until the roughly-every-other-talk discourse Alan gives about how “we live in a pink world and then we have a deviant thought but then we kersplat it until one day in the shower we have a kerpow that launches us into a new blue plane of existence—but that idea like most is more likely to be mediocre or terrible, so we have to then come back to it with something like science so that we can discard terrible ideas” becomes somehow ingrained in the hiring committee and they start trading their “kerpows” and evaluating them and coming to a better process than this “pink world” we see with our rose-colored glasses.
Because without that sort of systematic re-invention, aren’t we just guessing?
[1] https://www.researchgate.net/publication/313878823_The_impor... . Side note: in tracking this study back down I got to see a lot of interesting takes on it but the most common consensus is that “we make our first impressions in that first 20s and then do not budge from them” which I do not think is warranted given the data. That is, while a significant correlation coefficient is observed and it is comparable to the correlation between different long-form interviewers, at no point is that correlation coefficient, like, 0.9 or 0.99 or anything like that, suppressing interpretations that these first 20 seconds determine the interview. For that matter if that was your goal you would want as a control any other 20 seconds in the interview—you would want to prove that it was the first 20 seconds that determined the overall interview impression, not that people were judging based on some je-ne-sais-quoi demeanor that is visible throughout the interview.
Perhaps we could interview different people differently?
I don’t know why we have this obsession with subjecting experienced candidates to the same whiteboard gauntlet as a new grad. In fact, after a candidate has a bunch of experience you’d better be screening for other factors, since the role of an experienced developer can/should be dramatically different than someone with a few years of experience.
But no, still with the “challenges”. If you can’t do obscure puzzles with the same fidelity as you could six months out of college, it’s a no-hire!