The most important part is that the candidate can say "pass" and that you don't force them to squeeze out a shitty answer.
There is simply no better way to find out the breadth of someone's knowledge than to quiz them over a broad range of things. Think about it: this is exactly what engineers do with each other naturally and in a fun way over time.
"Hey, do you know about X?"
Compare this with Google's idiotic idea of asking a single question and judging the candidate on that randomly selected question.
No one who knows their stuff will fail to answer a long list of pop quiz questions well...but anyone of any skill level might not know the answer to any particular question.
Bloom's taxonomy[1] is useful here I think. Assessment ideally shouldn't stall at 'understand' or 'remember'. I do agree it is important to know about a large variety of things. The more specialized the job is, the more important it is for a candidate to do a decent subset of it on day 1. But sometimes you can explore a candidate's knowledge through 'apply' (use X to do Y) or 'analyze' and 'evaluate' (do Y), such as in a system design question, to see the structure of their thought process rather than something more scattershot.
Finally, regarding skipping a question, sometimes you can sense that a candidate's knowledge is limited in some area, which is a good time to just move on and try to go deep on something else. Not everyone has been afforded the same opportunities and has the same amount of experience. Software is a job where the potential to grow can do you better than a fixed skill set, especially if you see hiring as similar to making an investment.
[1] https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/
Please note that I have never been called back after doing that.
A genuine technical interview can and will be cut short immediately once it becomes clear that you are not bullshitting on your resume. The person who still asks that 20th question after you have already answered the previous 19 correctly is just being an ass, and playing stupid, time-wasting games with your interview.
That's the only time I've been that intentionally rude. He really deserved it, though. And (consistent with my other comment in this thread) I'd name them, but (a) they went out of business a long time ago, and (b) I can't quite remember the name of the firm. Something "clever" ca. 2004, missing a vowel.
Is it rude to do that in an interview? How am I supposed to decide if I want to work for a company without some insight into those things?
A: What does your competition look like in this space?
B: What's your plan for overcoming Company Y's huge lead in market share?
A: Tell me about your development process.
B: So what percentage of your process overhead is cargo cult edicts from upper management?
A: How much runway do you have? How long before profitability?
B: When the current owners look for their exit, how screwed will I be?
A: How do you deal with technical debt?
B: Show me the worst code in your repo.If the company focus on trivia that much on interviews and let juniors run the show expect no good senior working in there. That's never going to end well.
If 'getting the job' in that instance is the metric of success, then it doesn't work well at all, but in quality-of-life terms it's served me well.
Is it genuinely offensive to be asked the complexity of bubble sort? If such a question causes you so much offense that you lash out involuntarily, then you have some serious personal problems to work out.
Offensive, no. Just trite and tedious.
And more fundamentally, simply not indicative of the high quality, (genuinely) cerebral environment the interview subject, back in the original article, is looking for.
Why be rude to the interviewer even if that is the case?
I get that it's generally better to be polite. But when the same kinds of tedious and disspiriting (and sometimes downright condescending, and/or simply time-wasting) behaviors come down the interviewing pipe, over and over again... I can understand the temptation to bypass decorum, and allow a gut-level, emotional response to surface.
Being as while we might generally prefer to keep things professional -- we're not potted plants, either.
(I should note that, as a consultant, nobody ever asks me to whiteboard some code for them, despite often committing into their own repos and doing work that is no less important to them as a company. Almost like the hoop jumping is just some power-move crap on the part of the interviewer.)
If a few more people were genuinely afraid of being humiliated for blindly cargo culting google's hiring practices it might go out of fashion quicker.
At least, that's my hope. Which is why I usually tease people for doing this (yes, that upsets them).
I think you need to pause, take a step back, and consider who is actually making an ass out of themselves in that situation.
It's more effective when you're over-the-top polite when you set the trap with leading questions though.
Trolling gives me a bit of mild amusement in what is otherwise a dud investment of my time. The only people who seem to have a problem with that are people who have an exaggerated respect for authority figures (always unhealthy).
If, on the other hand, the interview process is obviously relevant to the job at hand and thorough I go out of my way to compliment the interviewer to their boss even if I know I failed.
Politely leaving without saying anything doesn't improve the situation for anyone. These pop-quiz questions need to be stopped.
Tongue-in-cheek aside, of course passive-aggressive isn't the ideal path to improving interview processes. :P
If you do not know such things then how are you going to take them into account when you have to solve a problem?
That's the point of the "hate" behind these questions.
Point one, it checks if you understand the most primitive sorting algorithm out there. Pretty low bar assessment of your general CS knowledge.
Point two, it checks whether the candidate has understanding of computational complexity.
Now you might argue that you don't need any of that in the day job, but that's on you. If the interviewer wants to check you have fairly basic minimum of understanding of very basic CS concepts, that's a suitable question to ask.
"Has memorized the complexity class for one particular, indisputably marginal sorting algorithm"
is equivalent to
"Has an understanding of computational complexity"?
Now you might argue that you don't need any [understanding of complexity] in the day job
That is quite clearly not what what said.
Just wonder what would you suggest as an alternative, if you need to confirm basic algorithms proficiency and understanding of big-O? You inevitably arrive to an algorithm question, and bubblesort is as good as any.
Actually the blank stare means "What is this, sophomore year again? I can't believe anyone still cares about bubble sort."
Just wonder what would you suggest as an alternative?
Look at their GitHub/Mercurial account (which you've been ignoring all this time in your desperate search for something to nail them on, but if you would spend a second or two looking, you'd find is chuck full of algorithm stuff -- much of it way more intricate than bubble sort). And ask as many questions as you like based on some project you find there.
Or, pick a problem you're working on that's algorithm-related (but which you genuinely don't fully know how to solve). Use that as discussion material. What approach they'd suggest, given that the data are sparse / not evenly distributed, whatever.
You know, as if they were a peer. Not an interrogation subject.
I like how you cast me into some mean soulless generalized interviewer and started bashing down your pain points, while I actually don't interview people. But a simple algorithm question is not out of line on a programming interview; it's not whiteboarding or take home assignments. It doesn't take much time to answer, it shows you have some very basic CS knowledge at least. I did not use bubblesort outside of CS101 class some 25 years ago and had to actually think about its runtime complexity, but it was not hard and it did not take long.
Again if you think basic CS knowledge has nothing to do with your job that's on you. Should add that I'm not really comfortable being grilled in any way on the interviews, it's not a very good dynamic. But I don't see any general, dignified way to filter out non-performers. Unfortunately just credentials are not enough in our trade.
Actually, yes. But that's just a matter of personal taste.
Again if you think basic CS knowledge has nothing to do with your job that's on you.
Hmm -- you keep coming back to this supposition, for some reason. And again, that's not at all what I was saying.
Should add that I'm not really comfortable being grilled in any way on the interviews, it's not a very good dynamic.
At least we're on the same basic page, then. The main difference I guess is that (1) I see the issue of maintaining "dynamic" -- the overall tone of bilateral respect, in the interview process -- as not just important, but very important, arguably crucial in fact; (2) dynamic aside, the "quiz-show" approach is rife with methodological weaknesses (for example, it highly favors those who cram on the material - or were simply lucky, and happened to have heard your questions before in there interview process); and (3) I find it quite easy to assess ballpark competence in tech folks (and to find red flags for incompetence) just from unprompted, regular discussion.
But again, that's me, not you. You can apply whatever filters you like. Like Steve Jobs said (in a different, but related context) ultimately either they'll work or they won't -- and everything will sort itself out.
As a software engineer, there are things I do all the time: Reading unfamiliar code and third party documentation, do some debugging (often based on logs), and try to write code that is well factored and thoroughly tested. I want to know whether any prospective teammate is good at those things for their experience level.
Interestingly enough, the general algorithm questions are easier the newer you are, and the less you know about all the topics I mentioned, because more often than not those are learned on the jobs, while the basic algorithms are learned in college. I end up reading papers on algorithms at work, but they are not algorithms that I expect to be general knowledge, and that I'd expect people to read on the job when they are applicable: How many candidates are going to be able to explain hyper log log, Paxos or Lamport clocks on command? Is knowing all of those three by heart a sensible pass/no pass signal? I for one don't think so.
And hence, kind of silly to use as material to grill people on.
Bubble sort question probes the understanding about the complexity theory, hopscotch hashing about concurrent algorithms and CAP theorem about problems with distributed databases.
Naturally when these questions are not open for discussion and instead only a short answer is expected then there is in my opinion something wrong with the interviewer or with the company.
You have to understand that the other side knowns in fact nothing about you and I find these questions to be quite fair to improve the situation.
It's just that, again, you're picking a very marginal example to do that with.
And more fundamentally -- simply asking someone "What's the complexity of $foo"? doesn't tell you anything about whether they "understand" complexity. It only tells you whether they've adequately memorized that particular cell on their crib sheet.
As they are thoroughly incentivized to do, thanks to people employing interview techniques like these.
Naturally this question in isolation is not very informative and it is hard to differentiate if the answer to it comes from a more general understanding or is just memorized. It has to be supported with other questions and wider discussion.
What I want to say is that in a larger context this question is still not unfounded and it is hard for me to consider this or any other listed questions as a signalling a dominant status.
I think that these three questions showed are quite good and fair (for example I did not what is Hopscotch hashing, but it looks quite important after looking it up, so thanks for this).
Some of such questions might be possibly indifferent for the position you were applying, but seeing some hate behind these questions is in my opinion quite unjustified.
This is absolutely absurd. I have a CS degree from Berkeley (universally considered one of the best CS programs in the world), and I spent the last few years on an unusually CS-focused team within Google (specifically, artificial intelligence). Off the top of my head, I could easily imagine not remembering what exactly bubble sort is. What's the use of retaining the details of an algorithm that's literally held up as the "don't do this" solution to sorting? The most I can imagine being useful is knowing the theoretical best case average complexity of a sort. And my job is wayyy more CS-focused than the average engineering job at Google (let alone elsewhere). You heavily overestimate how useful rote memorization is to real-world productivity.
Whether a candidate _understands_ these concepts, on the other hand, can actually be illuminating. If you described bubble sort to someone and asked him to describe the complexity, then you're getting all the signals you're describing about CS understanding and education. (though there's controversy about the usefulness of these skills as well)
Though to be honest, Google uses those kinds of questions and apparently treats them as useful, so I'm not sure why you'd have wanted to work there.
Eventually got fed up with their process, told the interviewer what I thought of it and hung up.
(which, to point out at least one upside, is the only way I've ever discovered of getting a company I don't want to work for to stop bugging me with recruiter spam)
Naturally if there is no option for the discussion when somebody does not know the answer off the top of their head then this would be really absurd and pointless situation.
This question is not good for selecting out somebody for their specific knowledge, but it is in my option a good one to prune out candidates that manage to demonstrate their general ignorance about the field.
Ah ok, I guess I misunderstood you. Though this "absurd and pointless situation" does in fact happen: There definitely is a class of interviewer who will expect off-the-top-of-your-head detailed knowledge (like being able to remember bubble sort's details from just its name).
No, it means you're smart and have learned to sift out unimportant matters of detail ("Was bubble sort, which I only ever had to think about for a week or two the first half of my sophomore algorithms class, and was only ever brought up as a negative example anyway, O(n^2) or O(n^3)?") in order to focus on much more interesting and useful stuff.
See, I am not a recruiter or interviewer, but this question feels to me like an useful one to prune out ignorant fools who start to lament how unfair is to ask such questions instead of giving more or less correct answer.
Naturally if short direct answer is the only accepted answer and there is no way to discuss these things with the interviewer then I would also walk away.
not knowing or remembering doesn't preclude one from understanding that one approach or technique can be more (or less) costly than another.
Reading the Aphyr "Jepsen" series is what helped me learn it... when I was still in high school. And this was necessary because I was curious about which database I needed to use for a particular side project I was writing.
It's totally possible to just Google "bubble sort complexity" or "CAP" and learn about it.
Yes, you can today easily find quite a lot of information and learn about it, but this was not the situation the original poster lamented about.
The claim was that when faced with such question, one can just google it and read the answer. Problem solved. Because these are just some random facts like who is the president of France.
My point was that in my opinion these questions actually probe the general understanding about the CS field.
As a matter of practical use, most of the time details about algorithms and data structures, from implementation through complexity analysis, can easily be regarded as "mere facts." Those times when this doesn't apply are times when somebody applying formal academic level CS is surely required, but these are far less frequent than most programming tasks require--even at Google.
This is like knowing that lookup complexity of the hash table is usually O(1) (but could become O(n)) and one should use it instead of list when frequent lookups are necessary.
I am not an interviewer, but I think that the questions listed were actually quite good pruning questions and if you do not know them, well, there are the search engines and you can look them up for the next time instead of starting lamenting about the unfairness and showing the ignorance that is in in this case in my opinion demonstrating the serious negligence.
Instead of asking about Bubble sort, ask what is generally the complexity of a naive sorting algorithm and how good it can get for a general case.
I have investigated immutable hash tables but I do not know much about concurrent hash tables. This is my blind spot, so I do not know what the relevant question about the Hopscotch hashing would be.
Instead of asking what is CAP theorem, ask if you can make a distributed system with consistency and availability at the same time.
I think that this was the intention behind these questions after all and is the reason why I am trying to argue in support of them.
When I'm trying to evaluate a candidate, I want people who proactively search out knowledge in that middle zone. People who understand complexity theory, and CPU caching behaviour, and the C10k/C10M problem just because they know they might need it some day. Because they want to work on problems where this stuff is relevant.
This sort of knowledge is the difference between hearing "our website is slow - I think its the database?" and "our website is slow - I took a look and our database server has run out of open file handles. But I'm still confused - that shouldn't explain the latency we're seeing."
The CAP theorem is a great example of this sort of middleground might-be-useful knowledge. Understanding how your database will behave in the face of network partitions is exactly the sort of thing that separates good engineers from great ones.
And yes, I too keep forgetting this name but I could talk about the issue if they explained what real problem they have.
As for bubble sort, I'd just write
for (i=1; i<n; i++)
for (j=i; j>0; j--)
if (X[j-1] > X[j])
swap(j-1,j);
And insertion sort actually is better. It can be made to work online, keeping all elements seen so far sorted before receiving the next element. And I think it ought to be more cache friendly than bubble sort.
1. This isn't the sole and probably not even the dominant reason for such questions.
2. When people resort to such intimidation you can out-alpha them by threatening to leave as if they were not worth your time (has to be convincing, ofc). Intimidation is a specialty of the insecure so this trick can work, though you may want to consider whether such job is going to be fun.
Doctors go through years of schooling involving a substantial amount of memorization as well as board exams, residencies and so on before they're allowed to do much doctoring at all. This is the case because it is crucial that they not forget in an exam that neck pain combined with a headache could be a sign of meningitis, etc.
If an engineer needs to know which sorting algorithm is best, taking some time to research and google it is not going to cause a problem. That's why questions like this are bad, even offensive--they make people feel as though they're bad engineers when actually they're being put through an ineffective interviewing process.
Why is it crucial if it's just a "5 second google search"? And not every doctor is dealing with serious life threatening illnesses, just like not all developers are working on bootstrapping their uber for cats with Angular 2 and pouchdb.
Because you can't google every single symptom. And sometimes the symptom is only mentioned very briefly in passing. And sometimes the patient is hiding the symptom. And sometimes the important piece of information isn't a "symptom".
So, you need to have these in your head so that you can connect the dots to "I thought it was X, but new data Y suggests it might be Z. I need to go push on abdomen, put stethoscope to stomach, right now while patient is sitting in my office." A good example of a non-symptom that would go into your diagnosis--patient just got divorced and started dating again ... hmmm, STD's are probably back in the probability pool again, might want to check for one given the other symptoms.
Where computers are useful, but, sadly, we don't have good implementation is spotting localized trends. If one person comes down with Karposi's Sarcoma, that's an anomaly. If 10 people come down with it, that's a problem. It would be good to have databases with data from multiple doctors so that we could let the computers spot those things.
Not every illness is lethal, for most of them you can just make it worse. Even less requires quick decision. Also, software evolves and scales much quicker than humans which makes our industry less standardised.
> That is not something you can say with someone's code being written this exact second. There is always time to research, test, experiment, and change.
In my world, there is a risk assessment which defines if we have time to research, test, etc. or just fix it on the go, because it is not a big deal if it occurs. In other cases we have a cold sweat and dead silence during deployment, don't we?
Not sure why doctors don't look after their patients. Is it because there is no incentive to? If they do bad diagnosis that means the patient comes back and the doctor gets more money. As long as they don't get sued there's really no harm to the doctor.
As a software engineers we are incentivized to perform well. A crappy system means time spent debugging issues, money lost for our employer. We also take pride in building good systems so we can take that experience to the next company and get s higher wage or position.
As it is, we can't even seem to find a program that can analyse an ECG better than an overtired and exhausted resident at 4 am, which is really quite a low bar to cross.
So, let's take heart attack diagnoses in emergency rooms--a fixed procedure laid down by following statistics is WAY better than human judgement.
Generalized abdominal pain--computers aren't so good. There are a lot of possibilities and most of them are about equal. The doctor is more likely to tease out the problem over time through conversational randomness.
Some obscure genetic disease--a computer is going to be FEROCIOUSLY better than a human. A computer can keep all of those .00001% options in mind as tests and diagnoses rule out the far more common problems until the low-probability things pop up into statistical relevance.
Also, it may surprise you, but there are many highly experienced and extremely capable programmers that didn't study computer science in college. Filtering those people out because they can't answer programming trivia questions is, at best, hugely short-sighted.
It's called being flexible and able to learn, and it's the single most important thing you're supposed to get out of your formal education.
Bittorrent is a great example. All the pieces had existed for a while, but Bram Cohen put them all together into one place. After that, lots of people created other BitTorrent clients. However, without that first, smart person, it wouldn't have appeared.
He finds that such trivia quizzes are strongly biased towards supporting younger candidates who are fresh out of college.
I mean, yes, it's certainly possible to be ignorant about it, lots of people are. But unless you somehow want to hire people who are completely uncorrelated with being good employees, the distinction between unwittingly and wittingly being a terrible interviewer seems a bit thin.
And the distinction isn't thin. It's the difference between an easily solvable problem and a really hard to solve problem.
But how else is the interviewer going to prove how smart they are, and signal their dominant status to you?