I observe that anglo-saxon peoples teach their kids to voice their concerns, needs, wants, and opinions, and to expect (!) results from this. (Compare how american kids behave at airports, train station, and public places with kids from other countries, e.g. french speaking countries).
This helps them approach situations in life with a healthy dose of self-worth and the baseline state of mind that if something is wrong it's not their fault.
In contrast, many other cultures teach their kids that it is impolite/crude/disrespectful to bother adults with your half-baked thought, or (God forbid) unexpected wants, or complaints.
I come from a culture where you are expected to moderate your verbal output when addressing your superiors (e.g. teachers, boss, etc.). You speak out only when you have something worthy to say.
Example:
I had trouble going to "office-hours" when at university in North-America. I always got the feeling that I am bothering the prof, and it is my duty to study more to understand the material, and I should not take his time when I merely did not understand something.
My anglo colleagues seem to not have such qualms. Their thinking is more along the lines of "I don't understand (not my fault), it is his job to teach me (most likely his problem), he has to explain this better, I pay for tuition after all...".
I am happy to have unlearned some of the above things.
But generally, I find the speak-your-mind-at-the-white-board-while-we-watch interview tailored for cultures where kids are taught to be vocal, and are difficult for equally intelligent people from other backgrounds.
On the whole the West tends to have lower power distance than the East. And Anglo and Germanic countries tend to have lower power distance than Romantic countries, while Scandinavia has the lowest power distance anywhere.
France has quite a high power distance by Western standards, although it's still lower than most Eastern countries.
The US tends to have rather high power distance by Anglo/Germanic standards although if you look at all the cultural dimensions the US tends to be an outlier towards moderation in its cohort probably due to its amalgamated culture (the US is hardly a strictly Anglo country).
http://www.clearlycultural.com/geert-hofstede-cultural-dimen...
I believe this is exacerbated by the sorts of cultural differences you highlight - I'm (relatively) loud and brash, many of them are reserved. I have no doubt there are lots of signals of confusion/understanding that I'm missing, but I don't know how to find them either.
I recently got my mid-term evaluations from the students (anonymous) - I was shocked to see many of them spent (or at least claimed, which I have to trust in absence of other data) 11+ hours/week on the work outside of class - triple what I expected. (and the answers formed a believable curve of results, so again, I have to trust it absent other data). I had only one student up until that point say how much time they were spending. Everyone else was silent.
Engineers really should communicate clearly about technical stuff. People solving complex problems really should concepualise their approaches clearly, ideally so that can be expressed in words. People really should reach out for help when blocked.
And if Anglos find it easier to do those things, then good on 'em.
I understand your comment isn't advocating for what I am insinuating, but in general, I think the sentiment that companies need to adjust their culture for their candidates is problematic.
There's a lot of variation within Anglo-Saxon cultures too. In New Zealand, we're usually a bit more reserved, but I can see that we're more forward than in many Asian and Pacific cultures. Also people everywhere have variation in introversion/extraversion and in their self-confidence.
It would be great if there was a way to do hiring that could identify good candidates without bias among these different things. Probably the only good way to hire is to simulate the real work environment as closely as possible. That's difficult and expensive though.
The best I can think of is to have a 45-60min written test with technical questions/problems going from very easy (where failure to solve is a red flag) all the way to more moderate, some difficult, and maybe even one very hard problem (solving which is a green (?) flag).
No need to be in the room with the person. Can put a webcam to record test if cheating is a concern.
After the test, the candidate gets a 5-10 min break, a drink, while the hiring team of 2-3 people take a quick look over the result of the test.
In the second part of the interview they all go into a room to have a chat based on the problems in the test. The candidate gets a chance to defend his choices, show additional skills, while the hiring team gets to ask tangent questions (how would you speed this up? why this data structure? if space was at a premium how would you improve this?).
I think the secret is to adapt the tone of the discussion to keep the candidate on the edge but avoid intimidation. If the candidate is a confidence artist, test him with more and more direct questions that he cannot weasel out of. If the candidate is more shy, engage him/her with a brainstorming like discussion, and go of on tangents from there.
The written test plus the opinions/notes of the 2-3 people present at the discussion should paint a decently detailed picture of the candidate, and also provide a written record of his/her performance for reference and comparison with other candidates.
Test results probably do correlate with job performance, and it may be better than an interview (not sure) but it's say it's still quite a long way from giving an accurate answer.
I sought help from my professors and teaching assistants in college not because I felt it "wasn't my fault" or even that "it is his job to teach me" or that "I pay for tuition after all" but rather my thinking was, quite simply, "I need to learn this and I'm finding challenges with the current approach I am taking... maybe there is another way" where "another way" just so happens to include pursuing some direct face-to-face help from someone else. The line of thinking was outcome-oriented (I need to learn this/accomplish X) and I'd do whatever it took to make that happen.
That kind of discussion between co-workers happens all the time. You suggest ideas. Have you tried this? Yes, but it has that problem. Oh, so what about that other approach? And so on.
It's a lot less stressful than an interview where you are expected to find the right answer, and it's also much closer to your real job.
I think the idea of modern open office is built on the same idea that people would be thinking, communicating and collaborating at the same time, but I hate open offices for this precise reason.
What I'm saying is that there's an underlying fear of looking bad when spouting off your mental rough drafts, during an interview. What would be helpful if interviewers were clear about their expectations, as in, "You will be graded on your overall approach, not your initial ideas." or "We don't take points off for hints."
How else will you prove that YOU shipped those things over that period?
* Github account? Ha, my code is on github in many places, mostly without attribution, none of them uploaded by me.
* Recommendations? Ha. Unless the person recommending you already works at the company, their trustworthiness is zero - they could be a paid accomplice.
* Resume? Unless you were the sole employee, there is no way to tell how much actual work YOU did.
* "Tell me about that one time you solved an interesting problem?" Ha. Can easily be a well-practiced lie.
(yes i have seen all of the above)
Turns out that the only sure way to tell if you can do X, is to make you do X and watch you yourself actually do it, in person.
Now, I am not accusing YOU of fraudulently trying to get a job, but people do. An in person interview is a good way to see.
(as always, i speak for myself only, not my employer, my cat, or our Great Lord FSM)
I told the candidates that they did not necessarily needed to show me a working application (although that would have been great!), but simply to document what architecture, language and approaches they would use to implement the task.
Also the task could be documented in a few hours, and those that got close to coding a few classes might have spent a day at it. Now the argument could be made that I did not remunerate this people, but keep in mind the work they did was not expected to be more than a few hours.
For me, this is an ideal technical interview. Also, I ended hiring a candidate though this method, and he is by far the most productive and creative force in our team.
I agree if that was the only thing you asked, you wouldn't get a good answer. However, if you then dug into their answer you can probably tell if it is BS or not.
I remember having this exact experience with a junior dev who claimed to worked on some interesting projects at Uni, but when you dug deeper, you realised he knew very little about the technology he supposedly used.
That said, my company isn't really interested in hiring people who can code an algorithm to balance b-trees. My company just need people who know how to ask questions, write maintainable code, and use a profiler.
Also, how common do you think these people are? IMO, not common enough to design an interview process around. Then again, I don't subscribe to the "one bad apple will torch the orchard" paranoia popular in this industry.
I don't write code on whiteboards for a living, nor am I applying for a job that will have me coding on whiteboards, so watching me do that will tell you almost nothing about the job you're actually hiring me for.
I interned successfully at FB, Microsoft and Google as a SWE in 2008-2010 and interviewed with a lot of other tech companies (not always but often successfully). Like you, I tend to freeze up in a whiteboard situation, and have most of my life, going back to college and prior (I studied in Romania where at the time you had to stand up and answer questions and use a whiteboard to answer them). I find that a few things help:
1. Practice practice practice. Simulate the interview with a few friends multiple times. It gets easier no matter how shy you are.
2. Smart interviewers and companies open with easy, friendly questions that put the interviewee at ease. It's a sign they understand this is stressful for everyone, no matter how confident, knowledgeable and smart they are.
While these interviews aren't perfect, with the right attitude they are helpful: it shows how the person thinks through a problem and communicates their ideas, and how they "work together" with the interviewer.
These are important skills that are important for all people regardless of their title.
I also believe a solid CS foundation is important even if coding is mostly putting things together. A systematic, mathematical understanding of complexity of Algorithms and Data Structures shows in one's everyday decisions.
There are only a few valid alternatives in my mind: 1. Take-home project work More realistic assessment of how effective a person is. Really good simulation of that, but no signal about their collaboration style, interpersonal dynamics etc. 2. On-site trial for a week or two This is probably the best way to assess someone's ability - internships and residencies are as close to real life as one can get. It's hard for people who already have a job, and expensive for a company to do. Simpler proxies/filters are needed.
Am I missing any other ways to assess candidates?
Yes. You are completely glossing over the fact that the candidate might be bringing something special to the table.
You have literally not left any room for the candidate to showcase what they've learned (again, not minutia of algorithms but something completely different..like the candidate could be a specialist in how compilers do register allocation)
How about the interviewer improving their interviewing skills to be able to identify strengths rather than being a pseudo automata that docks points?
Additionally, explaining how a system works is valuable, and I captured it when mentioned communication abilities.
The general process is fucked up because the interviewers are incompetent enough that they can't tell who is actually a smart guy. This is why incompetent interviewers resort to binary decision making on programming questions.
Here's a way to think about it. Let's take two candidates with equal communication skills:
1. This candidate solves all coding problems fast, with great syntax, good style and discusses algorithmic complexities well
2. This candidate is smart, can go deep in their specialty (because they've spent years developing complex expertise) but couldn't do a medium-level tree traversal without hints or has bad coding style (on whiteboard)
Who do you see getting picked (at Google)?
I remember a friend of mine at Google (a Java programmer) docked points off a candidate (a C programmer) because the candidate passed length of array as an argument. I was like wtf? Seriously? You don't know C?
The data-driven committee-based process completely ignores that not interviewers are equal. Their feedbacks on paper are not equal.
There's a reason scouting still exists in Baseball/Soccer/Football because data-driven process has given mediocre results at best. Interviewing is much like scouting. You cannot use mediocre engineers to identify who's a real talent.
"Real talent" is innate and thus no amount of outside valuation will ever do it justice. No one else, but the "real talent" holder, can effectively judge his "real talent" so it is up to him to make it known to others that he does, in-fact, hold "real talent."
The process is designed to find people that are good enough to do an array of jobs. "Real talent" gets scouted, because it's shown that it "has got the right stuff."
This argument boils down similarly to the "smart but lazy" exasperations. "I'm talented, but no one realizes it!"
Candidate 1 is getting the job at my startup and Candidate 2 will be getting a referral if he ever decides to take up a teaching position. Deep specialization is useless unless you're doing consulting (in-house and out-house) or teaching.
They're both valuable, perhaps for different types of work.
A balanced tree does not tell me much about a person's knowledge but how they are able to apply and learn CS concepts does. Deep knowledge of a problem domain is valuable, but also highly perishable - the world changes fast and good engineers need to learn and adapt. Furthermore, if a candidate speaks about a domain they have deep expertise in, we have two more problems: 1. The interviewer might not know enough about the field to assess the candidate. (Generic CS algs and data structures are a good common denominator). 2. A candidate's knowledge about a field doesn't tell me how good they are at reasoning. They may have spent 5 years doing that one thing - that's enough time to learn almost anything in a limited domain - say OS schedulers.
How would you structure an interview process?
-- Furthermore, if a candidate speaks about a domain they have deep expertise in, we have two more problems: 1. The interviewer might not know enough about the field to assess the candidate. (Generic CS algs and data structures are a good common denominator).
So the problem is with the interviewer and not the candidate right? The interviewer is unqualified to interview this person
2. A candidate's knowledge about a field doesn't tell me how good they are at reasoning. They may have spent 5 years doing that one thing - that's enough time to learn almost anything in a limited domain - say OS schedulers.
Couldn't the same be said of candidates good at whiteboard tiny algorithm puzzle solving? They may have spent 5 years doing that one thing - say algorithmic puzzle solving. The fact that books and coaching sessions industry exists should prove this right? Your process doesn't give as much weight to someone who has the passion to go deep in something productive as much as passion to solve toy algo puzzles
Re experience - yes, if you're interviewing for someone with deep Java or ML knowledge certainly make sure the interviewer can assess that.
Logic puzzles are not perfect - I used to be really good at solving them and if I ever have to interview on them again, I'd definitely have to practice first, to get up to speed. But I do think that being able to solve them is a meta skill. If you get really good it entails that you are good at reasoning and solving complex problems, or at least a specific class. If you are good with trees you'll likely be able to answer a trie question. It doesn't mean you're just good at that, but that you can apply knowledge to new problems - which is exactly why you are hired. An equivalent real question I got in Java was how to access a private member of a class. If I knew the answer it would mean I have some random bit of trivia in my head. If I solve it using Java knowledge it means I can apply my domain expertise in a new way - that's compelling proof of problem solving skills. If I ask why would you want to do that, it indicates reflective and critical thinking skills.
A seasoned engineer in ML or Java should, imho also be able to solve puzzles. They are, in many ways, better proof of abstract thinking abilities and problem solving skills than specific knowledge. But I agree they are both important - if I want to build a search engine I prefer to have a seasoned ElasticSearch lady lead the project than an equally smart person with no background.
Re your friend, if it helps, I interviewed at a lot of companies and probably failed about 70% of interviews in the first week or two. I got much better after a few weeks, which proves your point that interviewing is also a skill. And I mean for both tech and behavior questions. That's normal, I think.
Re Google, while I don't represent the company, there are public statements that they'd rather have a false negative rate rather than false positives. It's better to say no to a great applicant (politely) than yes to a bad one. If you've ever had to let someone go, or deal with a bad team member, you know what I'm talking about. Of course there's the occasional interviewer who asks a silly question or gets something wrong. Do you never make mistakes? It's impossible to ensure everything goes perfectly in such a large company, but I can tell you, as someone who works here, that people try really hard to do a good job at it, and leave everyone with a good impression. It's the rational thing to do: if you treat people poorly 1. You're and asshole and 2. They'll hate you. Nobody likes assholes especially at Google, and I can share that I haven't met one yet. That's pretty amazing given the size. Thanks for engaging in this discussion, I really enjoy it.
Let me foreword by saying that I am not against coding interviews in general. But, I am against the interview process as it is implemented NOW. As implemented today, the process is overweight on coding optimal solution with good coding style.
Translated to humanese: "It doesn't matter if you invented linux kernel at your last job. We can't hire you because you didn't solve this tree problem in time or had bad coding style as COMPARED TO OTHERS who don't know shit but can code puzzles on whiteboard"
What I think an ideal interview score looks like:
Score = w0(whiteboard-coding skills) + w1(algorithms) + w2(area of expertise) + w3(design) + w4*(lets work together to invent something new/passion for engineering)
As it stands, the interview process has the following weights: w0=49%, w1=49%, w2=1%, w3=1%, w4=chunk change
See what I did there? The way interviewers in these companies are hiring is by calibrating their favorite coding puzzle over a wide range of interview candidates. They simply don't account for the fact that that person might genuinely be good at something that the interviewer doesn't even know exists.
>> A seasoned engineer in ML or Java should, imho also be able to solve puzzles. They are, in many ways, better proof of abstract thinking abilities and problem solving skills than specific knowledge.
You keep asking for proof that the candidate should be awesome. Where's the proof that the interviewer can adequately judge based on the equation I gave above? The candidate could be seriously good at ML but if you send the average java coder at Google to interview them then gosh, how would they even recognize his genius? Now you'll say we have 5 rounds to eliminate biases. Buuutt, the way the hiring committee reads interview packet is, and I quote you:
"It's better to say no to a great applicant (politely) than yes to a bad one"
Which means, even if two interviewers think this awesome ML guy is GREAT, the candidate wouldn't be hired because the other 3 interviewers were incompetent and marked the candidate no hire.
What I'm trying to say is that the hiring committee based process encourages actively removing smart people and selecting the ones who solve whiteboard puzzles. The candidate could be a shit engineer in person but as long as they solve code puzzles in those 5 hours they get hired. Make sense?
Look, I'm not saying we need to eliminate coding rounds. All I'm saying is balance it out. A person who can solve a tree problem in one round surely doesn't need to be asked another stupid coding problem (which is asked just to make sure some random crackpot didn't squeeze in through the cracks). I mean, if you don't have confidence in your own interviewers that you need to keep asking those problems, what does it say about the process? That you don't trust your own interviewers? Isn't that the real problem?
You can choose to ignore all feedback I give here but then, I'd specifically call Google out and say that they're not a great engineering company until they figure out that their hiring is broken and they need to stop having their mediocre engineers keep rejecting the good ones.
Or...if you really care about working with great peers, show this answer to hiring committee and see what they think. They need someone to tell them, "Hey, why are you implementing a process that self selects mediocrity"
P.S: Why do I know it is mediocre?
1. Other companies have tried the same process and it didn't translate to anything
2. You know there are enough mediocre engineers at your workplace :)
I'm pretty sure the weights vary depending on roles: interviewing a tech lead for ML requires deep experience in That field and will have more of those questions than basic CS algs. Or at least a higher weight than it would in a fresh graduate.
It's not my place to judge the average ability of Google engineers. I'd guess it's a normal curve somewhat skewed to the right since the bar is really high. In my limited experience they're on par with top silicon valley engineers but have a different profile than startup ones due to self-selection.
The point of many interviews is to reduce false positives, both for skill and professionalism. It works amazingly well. Every single person I talk to has something interesting to say and is at the very least competent at what they are doing. Not everyone feels brilliant but everyone does seem bright. And I'm not drinking the cool aid.
Hiring good people is easy if you just test for skills. It's much harder to make sure someone is a genuine person, or at least not an asshole. Google does a tremendous job here and I worked at enough startups and a few large companies to know how incredibly difficult that is.
That other companies failed trying to do this can be because of many reasons, from bad engineers to start with, lack of resources, mistakes in assessing ability or rush to get devs on the team asap. Again, not my place to comment.
Every company asking its employees to interview prospects implicitly assumed they are good, or at least competent. If they don't, they have a bigger problem to start with.
If they hired someone they already think they are good, right? And if they ask them to interview someone else they think they are able to judge, or don't understand that logic (In which case they have even bigger problems).
How does a mediocre engineer fail a stellar one? If the candidate is really better than the interviewer, won't they be able to solve anything they ask them to?
I think the reason multiple interviewers ask CS questions is to avoid the issue you mentioned earlier: asking one question the candidate just so happens to know really well. Random sampling should solve it.
I interviewed at Palantir a long time ago and was asked super hard CS questions over the phone and in the first rounds and I killed it. The last round I was tired because I hadn't slept well and tanked one problem. Just one, everything else I had solved really well. Yes I thought it was silly to reject me and wish they had given me another shot. If you are at Palantir and are asked to interview someone, think they are bad, and they get hired, it's demoralizing - you just wasted your time and the company doesn't value your opinion. Sure, you could have a score and average things but again, firing is way way worse than saying no to a good candidate who just had one bad question. And everyone I interviewed with was legit. (I had another series with them later, for a different role and got an offer :-) )
I agree that weights could be adjusted but 1. How do you know that would yield better results? (Show me the data - I don't think anyone has it) and 2. It can also be a cultural choice to bias more towards CS skills vs design or whatever else. Perhaps the company has great design patterns and can teach those skills v effectively. I'm not saying it's true, just an example. If you had a company and ran the interview process your way, I promise there will be someone just like you who vehemently disagrees with your weights :-). But if you could prove that process A vs B yields better outcomes, most companies would definitely follow suit (if they pay attention, are good at adjusting their processes etc).
I don't actually know how they read packets, I just know they work really hard to avoid false positives, and I wholeheartedly agree.
Btw I was rejected by Google twice before - resilience is part of the game ;-)
The painful part is, I used to teach CS-101 and CS-102. I assigned that exact problem to my students and have graded several hundred solutions to it.
Now I warm up on lots and lots of coding problems before I start an interview process, but I always worry in the back of my head that I'll freeze again.
Ah, the memories come rushing back. I once froze in an interview, and force myself to blather something inane so that the phone wouldn't be silent. It went horribly.
I called back later that day or the next (it's been a long time), and basically said "that's not me. I know what I'm doing!" We had a conversation about the subject that I botched. It went well in part because I had already accepted that I'd lost that job; I just didn't want to leave an impression anywhere that I was a loon.
I got that job.
Had they not heard of Fizzbuzz? Or reversing a string?
Freezing can happen to anyone on any problem.
White board interviews are supposed to be interactive. When I solve a problem, I always verbally say whatever is going on in my mind and share it with them. This does two things : 1. Helps me focus on the next steps instead of letting my mind race through different solutions, context switch often and lose track of my own thoghts. 2. Gives feedback to the interviewer so he knows I am on the right track and if not, give me clues (with or without him wanting to) that I can pick up on and proceed further.
In the end, my white board interviews always went like a conversation, a deep discussion about a problem that involves coming up with a solution and attacking it to see its pros and cons. I have come to appreciate the process itself because it tests several things like communicating to the team, problem solving, being open to other ideas etc.
Edit : I'd also like to note here that I didn't get these skills in my first interview. My first several interviews went horribly wrong like you mentioned. It is a skill that you need to invest time in. And you have to believe that the time you invest in this will pay off eventually. It did for me in several ways.
I interviewed once with a couple of owners in a restaurant. They asked me something relevant, and I all but froze. I knew the subject, but it just would not reveal itself to me.
I candidly admitted "brainlock." They were very gracious about it, I soon got through that subject and we moved on. I was eventually hired.
The lesson being, for myself at least, not being able to produce "on the fly" is sometimes compounded by being afraid to show or admit a problem. Good people, however, will be forgiving and encouraging. They've been there themselves, we all have.
There have been a number of times, over the decades, that I have basically admitted ignorance or temporary brainlock, and it has always gone well, whether I was hired or not. I think, if nothing else, that it's refreshing and novel for the interviewer.
It really reduces stress if you allow yourself to be human. In turn, that's much easier if you don't put yourself in a position that "I must have this job."
Next time it happens, tell the truth.
"Oh, hey guys, I get really, really nervous about whiteboard coding. It's really scary for me and right now I feel like my brain has stopped and I'm going to fail miserably. Would it be OK if we do this another way? - maybe on a computer, or maybe I can just talk about it?"
If you need it, you can also say "Can I please have a 5 minute break to get my head straight? Maybe a quick walk to the bathroom and a glass of water?"
I would be utterly and completely shocked if a rational, intelligent human being denied you that. And if they did, would you want to be working for/with them anyway?
Please replace these guys with HackerRank. It so much better than this.
Not a word is said, they are clicking at their laptops, and staring at the whiteboard, as waiting for the genie to pop out of a bottle. All the while your mind is frozen and stuck in a bad loop.
This lasts an hour, you are barely able to complete parts of the problems and are frozen. Of course this affects your usually creative and sharp mind.
The torture lasts an hour, time's up! You shake their hands, as a kiss of death, and head out. As you are walking back, all the answers to all the problems they wanted you to whiteboard, come rushing like a torrent in your mind. Too bad, another 'botched' technical interview.
This is my experience as a battle tested developer who is shipped many products and has been programming for the love of computers since the age of 12 (professionally for more than 15 years). I am not going to be working at Google any time soon (not that a Google job really matters to me).