Again, I don't necessarily disagree with that, but I think it's always useful to consider how much of my interview criteria can be reduced to "do I vibe with the candidate" and make sure there's more to my interview than just that (there IS a place for it though. The world isn't an objective metric)
What I try to do is I try to learn specific things about the candidate not because they can be measured easily or objectively but rather because I think they are important.
I am not "vibing" with the candidate (although this is definitely a plus!) I have a checklist of things and I try to make sure, by the end of the interview, that I can run through my checklist and have an answer for each point on it.
> Can the candidate recognize and make use of an advice?
I must admit that following those two criteria helped me make the most effective hires.
I used to ask a question that was mildly confusing, but only to the kind of people I was trying to screen out. The people who I was trying to hire would usually get the question right away.
Basically, in one of my questions I had developers use a lesser-known threading API in C#. The people I hired either got the answer right away, or could quickly adapt with my advice and answer the question.
Another interesting anecdote: I used to ask candidates to discuss tradeoffs of the XML APIs built into .Net. The point was to discuss tradeoffs and to demonstrate more than superficial knowledge of serialization. (Most .Net developers have some exposure to its XML APIs at some point.) One candidate quickly hopped into a JSON is better than XML tirade. I ended the interview quickly.
Did you poke them for more? Did you ask how JSON compares to XML? If they only answers "Well, I've never used XML, but I was trained on JSON and won't ever use XML, I don't know any of the downsides of XML", then you were correct to pass on them. But if they responded "Because of reasons X, Y, and Z, JSON is better than XML, and all XML interfaces are equivalent in this way, so I can't really answer your question because while I know enough about XML to know that it's bad, I can't tell you exactly the difference between different interfaces, as I've avoided them." That would be an answer I would accept.
> got the answer right away
This is the major critique of asking these interview questions. There's a small subset of them, so candidates train on exactly them, and only know how to do these, and when you hire them they won't be able to actually code anything. Using ML terms, they're 'overfit'. Getting it right away is actually a bad thing, but then you get candidates who act like they are going through the process just to look like they are learning on the spot... it's a tough problem.
The candidate was incapable of doing so.
For the threading question, the reason why I chose an unusual API was for your exact reason: You'll never see it on a list of interview study questions.
I'm going to disagree here. Testing your code in small chunks can not only catch small errors before they propagate into something larger, but also give you insight into whether you are going down the wrong path.
We are not talking using external dependencies or frameworks. I am talking simple code that you just wrote in entirety, maybe couple of loops and conditionals.
If you can't predict what that small piece of code does, exactly, you are thoroughly unable to write reliable code, by definition.
Reliable code == code that does exactly what you intend it to do, nothing less, nothing more.
If you can't predict what your code does you are writing buggy code. Experimentation can uncover some of the bugs, but if you start with buggy code the experimentation will never remove all bugs -- only the ones that are easy to detect.
In general, I prefer the approach of writing out the entire solution at one go, and trusting that bugs will be easy to fix at the end if the code is clean enough to understand what is happening at every step. This is in part because there usually isn't enough time in interviews for running tests at every crux moment. However, I think our current virtual environment may make it easier to test for (and to utilize) these types of programming best practices, since instead of writing code on the whiteboard, candidates and companies can use one of the interview interfaces that allows running code as you write it.
Have they specified that you should be testing your code? If they haven't specified then it is wrong to expect some kind of arbitrary outcome.
People write code in many different ways, some do it very chaotically, some have very organized process.
I try to figure out which of these behaviors are fundamental to writing good code and which are not.
For example, I believe a habit of writing readable code IS fundamental to writing good code. If you don't have a habit of writing readable code, on average your code is going to be worse than if you had.
On the other hand, I believe writing unit tests IS NOT fundamental. Unit tests is one possible way of working with the code, but people were writing reliable code before unit testing became popular. Unit testing takes time and effort that can be allocated differently.
And so I try to look (I can't call it measuring because it is subjective) only at the behaviors that I believe are fundamental and dismiss others.
In the end, if you are smart person that can program well, you will be able to learn those other behaviors. But if you are not smart or you lack certain fundamental abilities, it is going to be much more difficult to fix it.
Maybe as do often the difference in this discussion is a quantitative one not one in principle.
You can sit there and plan all you want until you think you understand it, but you never actually know until you run the code. Being a good programmer is all about simplifying complexity. Testing your code as you go along to verify your assumptions is one more way to do this.
Sometimes this is fine. I do this when I execute external services because I may not be sure how they work exactly and I want to correct mistakes before I do too much design resulting from my flawed knowledge.
But when it comes to your code you should be able to tell if it works correctly. If you can't there is couple of ways to deal with the problem that I adopted over the years:
-- techniques that make it easier to ensure code works correctly are frequently also improving it in other ways. For example, making code simpler is important not just for reliability.
-- specifying your internal module boundaries and APIs is critical for anything non trivial. Abstracting parts of the problem is enabling you to solve each part of the problem separately. You can't keep 10k lines of code in your mind and tell if it works correctly, but maybe you can split it into multiple modules and make sure each works correctly, separately.
-- when I have written a module that I am just not sure works correctly, I refactor it so that it is easier to tell if it does what it should be doing.
-- make it a habit to write code that is what I call "obviously correct". One way to do it is to make sure it can never get into wrong state.
People make mistakes. You have to run the code. Do you submit PR's without running code? If not, then we're agreeing. Making code simple enough to reason about without executing is a great goal, but you cannot replace running code.
> You can't keep 10k lines of code in your mind and tell if it works correctly, but maybe you can split it into multiple modules and make sure each works correctly, separately.
But ultimately they need to work together, and that's the most important part. Again, you need to run the code here.
> when I have written a module that I am just not sure works correctly, I refactor it so that it is easier to tell if it does what it should be doing.
This is a good goal, though again, I hope you run the code. You don't know what you don't know.
It does not say whether it was specified well, but I think an application can reliably execute wrong process. There is no contradiction.
I don't want to say that making sure that your code does exactly what you intend it to do is at all simple.
As your application grows, as you start incorporating dependencies, external systems, code that came from others, things that you do not know very well -- writing code that does exactly what you intend it to becomes very difficult or even impossible.
But on the lowest level, when we are talking couple of variables, conditionals, loops, maybe small data structure -- I would expect a developer to be able to write it to do what is exactly intended.
Because if you have that ability and use it for every piece of code that you write, the overall result (even if it does not guarantee reliability) will be better than a person that cannot do it.
In my experience (and it depends a lot on our HR and candidates that are being sent my way) only one in about 10 candidates (for senior Java dev position) can do that.
And if they don't understand it, they're going to B*&^% and Moan about having to fix the bugs once they come back.
In general I’m deeply skeptical of interviewers coming to the table with their framework full of assumptions for how to evaluate the interviewee when in fact they have a solid chance of being the less smart person in that room and can’t get a readout on the smarter person.
My framework for the coding part of the interview is following:
* Does the candidate tackle the problem analytically? Being able to think analytically is critical to scaling from solving small problems to solving large problems.
* Can the candidate predict what his code is going to do before he executes it? Repetitively executing code to see if it works may seem innocuous, but really is a huge defect in candidate's process which makes it really difficult or impossible for them to write anything non trivial to be reliable.
* Can the candidate recognize and make use of an advice? Surprisingly, a lot of people can't recognize when they are given advice even when I directly give them solution to their problem. It is really disheartening to have a candidate dismiss your advice even when repeated multiple times. I frequently give advice to unstuck the candidate because seeing him/her work on the problem is more important to me than seeing if they can solve any particular part of the problem.
* Can the candidate recognize choices/tradeoffs that impact performance and/or simplicity of the solution?
* Can I have productive discussion with the candidate about designing his solution? Can the candidate converse in a clear and precise language?
* Is he/she caring that the resulting code is readable?
* Can they diagnose faults in their code effectively? Assuming they executed their solution and it is not doing what it was expected to do, I like to see the candidate follow some kind of analytical process to find out what is causing the problem. I even design the task specifically so that it is very likely they are going to make one particular mistake to give me that opportunity.
**
What I specifically ignore in the coding task:
* Whether the candidate knows the standard library. I don't care. During normal work they can look it up on google. On the interview I tell them that if they can describe the function I can tell them the name and how to use it.
* Whether the candidate can have flashes of insight. I don't get so many candidates to waste them on that kind high false negative signal. I have identified parts of the task that require a flash of insight. I give the candidate a minute or two when they reach it and if I don't see progress I give them a progressively less subtle nudges.
* Whether the candidate can do TDD/DDD/etc. It all depends on their experience. If they worked for non-TDD or non-DDD shop for a long time it is going to take some time to adjust.
* Whether the candidate can solve a very complex problem. The interview is already stressful enough experience for the candidate. I can learn what I need on a simple problem and I care much more to see the candidate's process rather than results. The problem is just hard enough to be out of grasp of most people at the very beginning and require some actual thinking. Some people get it instantly and then I am disappointed a little bit because I can't see them working on it.