The employer (or school, etc) wants to make successful hires and to avoid bad ones. False positives are always a big problem. False negatives are usually much smaller one. A process is good if it produces good hires, and no bad ones.
In principle, a process that hires 3 good candidates from a pool containing 5 good candidates and lots of bad ones is good. Avoiding bad hires is paramount. Missing 2 good hires is not as bad as making one bad hire, not near as bad usually.
Most employers don't care about false negatives at all. If they do, they probably care most about missing really good outlier hires, not mistakenly selecting the 5th best candidate instead of the 4th. The only reason to care is fairness. People instintively care about fairness so this affects practices but it's not nearly like false positives.
Employers that need huge volume (eg, the army) are the only ones that care about false positives as much as candidates do.
The candidate (student etc.) generally feels the process is "broken" when it is unfair to them. Unfair means that a person could have done the job well, but cannot get hired. Ie, if the candidate is a "false negative" then the process is broken.
Anyway... When things become systemic, the "brokenness" becomes acute to a candidate. If whiteboards (your architecture version, Google's CS version or any other version) are prevelant and you suck at them... you can't get hired. There are people who suck at your test (whatever it is) but would be good at the job. Your proxy is a biased against some people, every proxy is.
Like I said, a lot of things are like this. One side is working the numbers game, the other side isn't. Insurance markets, college entrance, executive appointments, PayPal's fraudster detection (bastards!), credit scores, investment (even startup investment)...
The problem is that very few companies know how much false negatives are costing them, so it is very difficult to evaluate whether the false negatives are—cough—a negligible tradeoff against weeding out the false positives.
False positives are right in your face. But it is nearly impossible to systematically keep track of the candidates that aren't hired to see if they are successful elsewhere.
So people lean towards eliminating false positives at all costs. It's another variation on "Looking for the watch where the light is better."
It's another variation on "Looking for the watch where the light is better."
These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position.
Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).
That applies in the case of caring more about false positives than false negatives, since nobody (or close enough) even knows how many false negatives they have.
Your statement that someone being successful elsewhere doesn't prove we have a false negative underscores the uncertainty involved. It doesn't invalidate what I'm saying in the least.
But if we want to be rigorous, we could do some direct testing. Let's say we give a telephone screening and four interviews to each candidate. Simpkifying somewhat, let's say that our policy is that a fail on any one eliminates the candidate.
If we hire lots of people, we can collect false positive and false negative stats by randomly passing someone who fails exactly one test.
If a lot of people who fail a particular test end up succeeding, we can infer that the test provides false negatives.
On the other hand, if everyone who gets the random "free pass" on one particular test ends up failing, we can presume it has low false negatives, and maybe abandon the random passes for that test.
The original joke/anectdote works because it leads to clearly suboptimal search - the guy knows his watch isn't near the streetlight, but he's looking there because he can see.
This is a very different situation. To abuse the analogy further, the guys best bet to find a watch seems to be just looking near the streetlight. After all, not only can you see better there, but most of the watches are near there too.
Your saying "false negative is only a cost to you if you fail to hire well for that position" could be easily disproved with examples of people, who were rejected and started their own competitive business (e.g. [1]). IMHO, one of the goals of having 10Ks of engineers on staff is to keep them from the competition. But since nobody is tracking this and the extreme cases like WhatsApp can be hand-waved away it's easy to pretend that the false negative's cost is miniscule.
The guy with the watch analogy only works if there is good reason to believe he’d be better off searching elsewhere. But we don’t appear to have that here....
Let's be generous and say the WhatsApp was a 10x unicorn so FB overpayed only 9B for it. Which makes 4.5B loss per false negative on each founder. If a false positive was, on average, 1M loss (which is an extremely generous in an organization such as FB in my recollection), it means you could have had 4000 false positives for each of these negatives and still come ahead.
And, it brings us back to the Type I and II errors - it's always cheaper to make one, which has a limited downside than the unknown (potentially very high one). Typical example given is that while you are on the way to an expensive cruise you realize that you might have forgotten to turn off the stove. If you go back you might miss your 10K cruise. If you cary on, you might lose you 1M house. On average, people who miss cruise in such circumstances fare much better than ones, who risk their houses.
I get what you guys are saying, there just doesn’t seem to be real evidence that is an issue nearly all cases. Whatsapp notwithstanding
So in order for it to make sense to spend that money, you want some evidence there will be a positive return. Far from contradicting my position, what limited data I have seen does not support that effort, at least in the general case. Certainly whatsapp and the like don’t speak to it - they are very much outliers.
By all means try it. If your false positive rate explodes it will be an expensive lesson, but that’s always a risk.
One last point, the idea that nobody thinks about this and try’s thinks is laughable , which is worth noting because some comments in this thread almost suggest that it’s true,.
With my example of sacrificing a 10K cruise for a chance to not lose a 1M home, I figure?
> Far from contradicting my position, what limited data I have seen does not support that effort, at least in the general case.
Indeed, the data does not contradict a statement "on average, false positives are 4.5B each", which nobody has argued. However, it does contradict the statement "false negatives have any cost only when you are unable to fill the position as the result", which you have made in this thread.
>One last point, the idea that nobody thinks about this and try’s thinks is laughable , which is worth noting because some comments in this thread almost suggest that it’s true,.
Again, it would be silly to say that nobody thinks about it. Evidently, a number of people think the same way you do.
At first, yes. But when one has had enough experience looking for scarce commodities (of whatever kind: genuinely attractive apartments, interesting music / restaurants, romantic partners, etc) one learns that the "where the light is" search technique only gets you so far.
Certainly in this scenario - the false positive rate is only important if the true positive rate is too low. It's not at all clear that is a factor.
That's the whole point -- in order to find evidence that it isn't working, you have to start looking at the false negatives --
or that is to say, where the "light is not".
LinkedIn could offer this as a service, if there was sufficient demand that they could monetise it. Every company will routinely check out every candidate’s profile anyway. They just need an automated way to check again in 6 months and see if everyone they reject is going to a competitor who is beating them in the market.
I would agree with you if the supply and demand weren't so skewed towards top developers. Do you think you are choosing between A and D people, or are you really choosing between C and D people and A and B people never walk through the door?
It's pretty hard for a hiring manager to be called out as too selective. However, if you let in a few bad apples then you look really bad.
The incentives are for hiring managers to be too careful.
It is, oddly enough, the case that if you are not hiring any bad engineers that you probably are being too selective.
Then again, probably one of the worst bad apple I have seen would ace whiteboard as he was quite good at trics. He just did not like the actual work involved in software development and tended not to do work he don't like except bare minimum.
Also you can fix some false negatives - allow people to interview again for other positions at the company. I've seen multiple people get hired on the 2nd or 3rd time around and ultimately be successful.
When we first got married my wife was really hestitant about me working on contract. Over the years, she's seen how much larger my checks are when I do.
Besides, there are some opportunities where companies just want someone to come in, fix a problem and then after that it's just boring maintenance work. If it's a contract, I can come in, fix the problem, and look for a more interesting full time job without being labeled a "job hopper". I can just put on my resume that the 6 month job was a contract.
I prefer contracting but I found that more companies are willing to pay me my minimum salary than the commensurate hourly rate I ask for to make up for the possible gap in employment, unpaid time off, and self employment taxes.
But, my wife has a stable government job with decent benefits that allows me to go back and forth between contracting and permanent and allows me to take risks working at companies that pique my interest without concern about the long term viability of my job.
People complain whiteboarding because they've had bad multiple bad experiences with hiring processes that involve whiteboarding and they came to a conclusion that whiteboarding itself is the problem.
Unfortunately, you can't fix a bad hiring process by simply changing the technique. Each technique has its own pros and cons, but no technique will guarantee success.
These are the most common problems I've seen with hiring processes:
1) The people doing the hiring don't really know what they're looking for. Instead, they go by a vague, handwavy notion of whether the candidate is "a fit" or not. You have to define what "a fit" means in such terms that all of the people involve understand it, agree on it and know how to test for it.
2) The people doing the hiring don't know how to test for their criteria. Whether you're using whiteboard or a take-home project or some other testing technique, you have to design the test so that it helps you observe the traits you're looking for. [1] And you have to actually train your people to interview candidates.
3) Lack of transparency. This is a huge problem and it infests all aspects of hiring. Some recruiters won't even tell you which company they're hiring for. A lot of companies misrepresent what you would be doing for them, sometimes intentionally, but usually not. Some companies won't give you perfectly reasonable information about the job. [2] And, thanks to the litigious nature of our society, no company -- anywhere, ever -- will give the candidate honest, detailed feedback of why they were rejected.
4) The company is unreasonably demanding when it comes to candidate's personal time. The typical two-stage process -- a phone screen followed by on-site interview round -- is demanding enough, because it involves more than just the interviews. The candidate has to spend time and effort on other aspects, too: multiple phone calls with the recruiter, preparation for the interviews, travel, and juggling all of this with their personal and professional lives. Some companies don't understand this, so they think there's nothing wrong with a process that involves a take-home project, a debriefing phone call, a face-to-face conversation to test for cultural fit and a full-day on-site interview round.
Most companies don't even know that their hiring process sucks. Of those who eventually realize something is wrong, many end up cargo-culting whatever they read or heard about "more successful" cases. As things are right now, the whole situation is eerily reminiscent of the fuss and hype the industry went through with software development processes. I wouldn't be surprised to see yet another wave of parasitic consultants pop up to prey on companies by flogging their hiring process improvements and best practices.
----------
[1] This is where a lot of people's complaints go wrong. The most frequent grumble is: why does this company's interview process focus on algorithms and data structures, if you'll just end up coding some services in Java? If the company is doing things right, they're using these topics as a tool to observe and measure whatever it is they're looking for; most often, it's the candidate's understanding of computational complexity. Just because you won't be writing your own implementations of common data structures and algorithms, it doesn't mean you shouldn't know how to pick the one that won't blow up your memory or execution time.
[2] I had a case where I was interviewing with a company that gave me a print-out overview of their benefits, which included the price of each health plan, but not what it covered; when I inquired about it, they replied they wouldn't be able to give me that information until I signed the New Hire NDA.
I've started to turn this around and so far our senior candidates have loved the experience. (I only wish I had come up with the idea.)
Instead of asking to write code on a whiteboard and solve an arbitrary problem, couple of our engineers came up with an inverted problem. And ever since seeing it in practice, I've been trying to introduce it to every interview I can think of.
We write a piece of code on the whiteboard, and then ask the candidate to review it, as well as modify it to better suit the problem and context. This way the question turns into a discussion on design, constraint exploration, service expectations and externalities. Hell, sometimes it even goes into deep language design issues. Sure enough, it still measures a level coding skills but puts a much higher emphasis on understanding what the problem and its scope really is.
It's also a lot of fun.
I work at Google, opinions are my own.
I think your assumption that Google interviews based on what you will do on the job is wrong. The reality is that for most SWE positions, we can't assume you will know the things you'd be using for the job because of all the internal tools. Thus, algorithm interviews are more or less a proxy for some combination of general intelligence and coding questions.
I'm not saying it's perfect and there are many things I don't like about it, but I think it's important to know what the goals are when making a critique of whether the process meets those goals.
It's a proxy for one particular style of coding questions: leetcode. It's nothing more, nothing less.
Calling it a proxy for general intelligence or coding in general is a joke. That's propaganda passed around the company to convince people that the practice should be continued. It strokes the employees own egos to think they passed some magic intelligence test so it motivates them to keep repeating that it measures intelligence/skill.
My assumption is that there are a lot more people who are studying for these interviews than you think, which is why the proxy is failing (or producing homogeneous results)
On the other hand, it's okay to target sufficient but not necessary. False negatives can be fine, so long as you are finding enough true positives.
Even if you are systemically missing out on a sub-pool who are good, it's not necessarily a bad thing - particularly if you are realistic about its limitations. To my mind the real problem isn't that you might miss good people (you will) but that if your filter is too constrained the good people you find might be too homogeneous.
IMO the biggest problem is that small startups copy the Google's approach while their hiring environment is completely different.
As an aside, if you've never done systems design or pseudocode on a Smartboard, you're really missing out. Copy/paste, duplicating a slide, recalling earlier slides, etc. Man, that's the way to design a system with quick iteration but still no worry of losing your work.
When I interviewed at Google, they asked me if my whiteboard code would compile. I said, "give me a laptop and I'll tell you". I don't think they liked that answer. ;)
Some of the biggest productivity gains I've had over the years were related switching from "writing code down" to "thinking in code" mentality and abandoning linear, paper-like ways of thinking, which hindered me when I was in college.
For anyone who believes that paper and whiteboards are "good enough" for coding, I highly recommend watching some of Victor Bret's or Alan Kay's talks. Pretty much any of them. They consistently touch on the subject of dynamic media as a way to empower people to... well, think better for the lack of better terms.
Not when you are literally writing code on it.
And there's nothing wrong with whiteboarding pseudo-code.
Hey if they work for you, that's great.
But the simple fact is that they don't fit everyone's brain. And that there are other tools available to accomplish precisely the same ends -- some of which have many advantages over whiteboards, in fact. Like these things known as "pen" and "paper", for example.
So this idea that they're part of "every engineer's" day-to-day work is, quite simply, disconnected from reality.
I do agree with you, however, that the problem lies not in the whiteboards themselves but in the way they've used. Which has truly been an epic fail.
> Like these things known as "pen" and "paper", for example.
When you need to share this pen and paper, you use a whiteboard. I don't understand this hate for whiteboards, which are a cross-industry cross-profession standard (including medical) when collaborating.
One of the more interesting things I have noticed is that happy engineers have a little free time each day. They aren't under crushing pressure at all times. To this end, it's very common for whiteboards to accumulate drawings and doodles. This is often turned into "the elaboration game" where one doodle is built upon (by a little or a lot) every now and then. This is probably because coding is problem solving, but also because it's a creative endeavor. Engineers who are happy tend to work around other happy engineers and those engineers are all creative with free time.
What's not to hate?
* unless both the board and markers are of very high quality, the boards tend to stain over time
* as do your fingers
* and the markers dry up, and the erasers get less useful as well (if they're available at all)
* you have to erase other people's crap (or wonder / ask if it's worth saving) first
* the dizziness that results having to turn back and forth between the board and your audience (that is if you don't end up simply talking into the whiteboard most of the time)
* and from stooping to pick up the pens / erasers / pen caps all the time
* and all of this, mixed with all those indelible memories of crappy interview experiences
I'll take pen and paper any day
which are a cross-industry cross-profession standard
Along with horrible interview experiences unfortunately.
Solved by getting a good office manager.
I'm not even joking. We have very large whiteboards in every room and in almost every space where people can wheel their desks in. Cleaners are happy to clean them up and make sure they are in good condition. One team recently complained that they may be too eager... (Our whiteboards are heavy glass, bolted to the wall. It probably helps.)
Office manager(s) on the other hand always keep their ears open and make sure the office has enough pristine marker packs and erasers available. Having a supply cabinet properly stocked is a cheap investment.
One of the most enjoyable things I can hear at the office is: "Let's hijack a whiteboard".
In theory, I guess.
But -- getting back to the original subject -- for some reason during interviews the odds that you'll get an inadequately stocked and clean whiteboard seems to be almost laughably high.
Consistent with the observation that in many places, the seem to almost make a point of the importance of positive candidate experience in so many other ways (from poorly prepared interviewers to inappropriate / irrelevant questions to stalling and ghosting, and on and on).
If that's the kind of whiteboarding you do, no problem. But, don't ask me to write a merge sort in C# (been there done that). If you ask me to sort a list in C#, I'm going to write:
sortedList = employeeList.OrderBy(e => e.LastName + " " + e.FirstName).ToList();
I get so hung up on this it’s changed how I ask the question. The last time I asked a sort question, all I wanted was a compound comparator. And even then I couched it as “we want the rest endpoint to support sort criteria so we can do a top five list, but to sell the idea, we are going to sort in the UI for now. Give me some code to do that.”
With R#
Generate Code -> create Relationsl Comparison and click on the properties that I want to use for comparison.
Every candidate I interview has the choice between coding on a whiteboarding, and coding in a text editor with rudimentary code formatting.
I really like it when people code in the text editor, because that's one less thing I have to transcribe in my interview evaluation.
Regardless of whether they will choose the whiteboard or the laptop, I have never, and will never mark people down for typos, confusing .length with .size, a missed semicolon, or any other petty bullshit that isn't pertinent to the problem. I'm not hiring a compiler.
AFAICT, the backlash is around interviews where candidates are asked to write code (or pseudocode) on a whiteboard. Being able to implement bubble sort on a whiteboard does not correlate very well with being able to deliver business value, in my experience.
To add to this, it's being able to write code on a whiteboard in a high pressure situation while talking about every step you're taking. Over time I've gotten more comfortable with this, but mainly by doing more interviews, not by getting better at my job.
I spent about 30 seconds looking at it, but at least the first chapter 1 seems to be about proper whiteboard techniques
To all the responders to my query, your answers are super helpful.
Being self-taught, there are just these few holes the internet doesn’t quite cover. But a little reaching out on HN goes a long way in filling the gaps.
Basically, if it is an idea, concept or design, draw it out. If it is functionally correct code that compiles, screen-cast and type it out.
The time before that was for a database migration where I mapped out each step that had to be done and in which order.
The point is not to have or not have the whiteboard during interview but how and what it is being used for.
The best way I found to interview candidates is to have few different discussions. One of the discussions I like to have is to try and solve some kind of problem that you could potentially have in real life. I am solving problems with people this way on a daily basis so this lets me observe candidate in a setting that is already familiar to me and hopefully to the candidate. The whiteboard is then used exactly as in real life -- as an tool to visualize and communicate ideas and hold it as a kind of cache and reminder for the duration of the discussion. The whiteboard is there during interview but I will not require candidate to use it. If I have trouble understanding the idea I may ask "ok, I can't visualize this, could you maybe draw it for me?"
Nobody programs on whiteboard in real life. If I need to program for presentation, for example to train my team on some topic, I will bring laptop.
The key about whiteboarding is it take away all the specific syntax and gets to the heart of the understanding the logic.
Thats what collaberation is for, i think many young engineers that only code with an ide are nervious whrn they dont have autocomplete available and they lose focus on the core logic.
Programmerd used to write on paper ... this entire process (coding without the ide/search engine) is lost.
But this is the part where the programmer shoes their strength in logic ... not syntax memory.
I feel like there's two angles to this; 1. understanding complex algorithms commonly used as tests. 2. understanding how to convey information on whiteboards efficiently, common paradigms, etc.
#2 is what I think I'd actually enjoy improving. #1 is just SAT style stuff that I infrequently use in my day job and serves as an annoyance more than anything.
I think this is largely the point.
They are terrible at compiling and syntax checking.
Theoretically you should already know whether someone can do the physical code part. That problem is largely solved for and don't require in-house interview.
In-house interview should be used for c̶o̶n̶f̶i̶r̶m̶a̶t̶i̶o̶n̶ b̶i̶a̶s̶ ̶f̶o̶r̶ a̶g̶e̶/̶r̶a̶c̶e̶/̶g̶e̶n̶d̶e̶r̶ determining if the candidate can think/design/solve for the domain space they're being hired into. A whiteboard is a really good tool for assisting in that.
They are optimizing completely different metrics than you do. They want crème de la crème developers that are perfectly trained not to think about certain classes of problems and waste time there but instantly come with acceptable answers, immediately recognizing the structure of the problem and having the ability to faultlessly transform it into a piece of code without a debugger. It allows the training from the best universities to shine.
The problem is when everybody starts doing the same, but without the prestige, salary, equity and lifestyle that Google provides.
Or without actually needing to problems that even remotely resemble the kinds of problems that Google (sometimes) solves.
I feel slowed down having to write out on a whiteboard when I could instead write some dummy code to get a point across. It just seems crazy to assume all developers everywhere must be good at white boarding.
I tend to move my hand faster than I can hand write, which is why I like typing for all forms of explanation.
Especially since I’m on a full remote team, whiteboarding would be really weird.
I'm sure a shared digital sketchpad would accomplish the same thing for remote workers.
The reason they are important is that they are part of every engineers day-to-day work. We use them to collaborate and design and understand.
Therefore, if a candidate can't whiteboard a design, solution, architecture, etc. in a manner that other engineers can understand it, they are going to struggle.
That being said, the way Google and other companies use whiteboards is an epic fail. Asking someone to code on a whiteboard is stupid. Asking them to solve algorithm puzzlers on a whiteboard is lame. This isn't how we use whiteboards every day so why should a candidate be expected to use them this way during an interview.