The annoying thing about refining interview techniques is you only get a small view of effectiveness: you can't tell how many times when you didn't hire it was the correct choice vs passing on someone great, so I can't judge from that perspective. That said, I haven't regretted hiring anyone that did well on this style of interviewing, without another coding interview.

I also think I can do this, but I believe most people (myself included) introduce a lot of bias when doing this. I've spoken to developers who in 15 minutes I might have written off as not very qualified, but after hours/days have realised are great at what they do, and very experienced in types of software engineering that I have had less exposure to.

I think we have to be careful when judging candidates with what amounts to "gut feeling" like this, as it's typically biased, and not aligned within the hiring team.

Since every interviewee gets the same set people then start to think this is more scientific than "gut feeling" - it is not, but instead of your own biases you have the biases of the template designer.

There are probably a few companies which extensively study if their interview process for the people hired (the only subset they have access to, which already is a problem) actually matches their job performance and then try to readjust the process, but it doesn't seem to happen very often.

This also gets us back to the old problem that we have no idea what a good programmer is or how to evaluate job performance.

I'd say it's not a bad idea to forward questions like that to the candidate in advance of the interview. That way, you avoid false negatives from people who just couldn't think of a good answer under interview pressure.

I never ask this sort of question because:

1. I'd struggle to answer it myself despite nearly three decades of fixing bugs. I could probably come up with some interesting ones after a few minutes of thinking, but most of the bugs that'd immediately pop to mind would just be recent bugs that were hard to explain without knowledge of the codebase.

2. There's no way to guide an interviewer on how to mark this question beyond gut feel. What exactly does a good answer to this look like? More importantly: what does a bad answer look like? Asking questions which are impossible to actually fail at, or for which the outcome is basically random and unjustifiable, is a very common failure mode for interviewers.

3. It tells you nothing about the candidate because they can easily just repeat a bug they read about on a blog last week. You can't check they actually debugged it themselves.

For instance I just searched for "most interesting software bug" and clicked the first relevant link, the second story on this Quora post could easily just be repeated verbatim to an interviewer over the phone:

https://www.quora.com/What-is-the-most-interesting-software-...

If the interviewer wasn't alert they might be fooled by it. Candidates will quietly Google for answers during talking questions whilst trying to avoid the interviewer noticing, and many other things.

That's why I emphasise that designing good interview questions is quite hard.

I agree that's not a great question. I usually have few simple questions that can tell you if your candidate really understands the platform they say they understand -- it doesn't take much.

I usually have one or two projects set aside specifically for new hires; the type of project that is maybe a month-long or longer but that doesn't really require any domain knowledge or big integrations. It allows them to have something to do when they're learning everything else. In the interview, I can ask them how they'd proceed with it and I always offer that I'm here to help them and will answer any questions they have. This is literally no different than what I would ask them on day one of the job if they got hired.

Is that more informative than whether or not they can reverse a linked list? I think so.

I prefer the even more casual, "What's an interesting bug you've been bitten by?"

What would be the downside to giving the candidate a heads-up that you're going to ask this question?

How do you know? Is this repeatable -- e.g. can you write down the questions and the criteria such that someone who is not you can follow them and come to the conclusions you would?

Do you ask all candidates the same questions, or do you go with your gut and hope there's no systemic bias?

How do you know? Do you check your interview assessments against actual performance among those hired (measured how)? What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected?

I'm pretty forgiving -- heck it usually takes at least half the interview before the candidate's nerves settle down.

Being a developer involves being able to explain yourself clearly. You can't just sit in the dark and code, talking to no one. If I came away from an interview with the impression that you can't code then I don't want you. The candidate has to do some work too -- they are there to convince me to hire them, not just pass a test and get a grade.

If you can pass a coding test but you can't express yourself in a way that anyone can understand, then that's a no-hire. If that's a cultural bias, so be it. I don't see how it can be any other way.

Now it's been my personal experience that culture doesn't matter too much. For more than half the team, English is a second language. HR will reject candidates for language reasons even before I get to them.

Not set questions, even, just a good conversation. What issue have they run into with Framework X, how do they refactor, what's the weirdest bug they've ever had to track down, etc etc etc.

SV seems to enjoy grueling 7-10 round interviews these days. I had one a couple months ago that was over 40 hours (of my time) and over 20 hours (of their time, that I was aware of). I just can't believe that much time provides 20-40x the value of a simple relaxed conversation.

I have found that this different animal is competent, and certainly can code, but is of the very unproductive, slow type of developer.

And so my biggest pain point is rarely around a developer's competence during the interview process (i.e. Does this person know the material and are they able to do the job?) but rather instead around their overall productivity, their "capacity" to build software using their competence.

There are plenty of people in the world who know the material but are extremely "slow" in actually getting things done and problems solved in a day job. Unfortunately, the in-interview coding tests don't do a very good job of measuring the person's productive capacity to get problems solved, only whether or not they can solve some coding problem in a 30-minute timespan.

I imagine I would ask for knowledge only practical experience could provide (interest in a topic is not the greatest teacher, but a decent one)

Better than fizz-buzz I think...

But you can never be sure either way. I would think that actually giving people a chance while defining quantifiable expectations would help here. Costly and time intensive perhaps, but I don't believe there to be a reliable shortcut. Especially not if the labor market is in need of more engineers.

No, that would be phenomenally stupid. You would do what teams actually do, and look at their recent playing history.

Note that this is also entirely possible for engineers.

How exactly? Most people cannot take their code with them and show off. It belongs to previous employers, and please do not begin with the "show me your GitHub" - it's unrealistic to expect people to devote their life 100% to coding. Just as I wouldn't expect a carpenter to build houses for fun in his spare time, I do not expect programmers to spend their spare time in front of computers. In fact I believe that socially it's a benefit if they have other hobbies.

And if someone turned up with code from their previous employer, that'd surely be a huge red flag anyway.

First, most programmers have at least some code they can show you. Stop fretting about the 5% who don’t. If you give up here simply because you have a theory about programmers “devoting their life to coding”, you lose.

Second, you call their references. Yes, really. I know it’s not “objective”. Do it anyway. You will learn a lot. Really.

Third, look around for other people they’ve worked with, who weren’t listed as references. Call some of them. See if you get different opinions.

If your candidate truly, honestly has no code available, no enthusiastic references, and you can’t find anyone who worked with them who will vouch for their skill, then do you really want to hire them?

Finally, yes: do a trivial fizzbuzz coding test to filter out the total fakes.

Listening to coders complain, you’d think that nobody else in the world hired anyone, ever. Let go of your need to make hiring a failure-proof system, and you’ll discover a bigger world.

Most junior programmers have code to show, and most of the time I don't use it for anything. I'd rather do some FizzBuzz level whiteboard that we spend some time discussing and extend on. It gives a far better picture of their future as a developer. I know this excludes some, but that's the trade off I'm making.

I call references every time, but calling around willy nilly is not a good idea. Apart from the fact that it's not legal, most of the time I get "Yes I've worked with X as a developer for Y years" and nothing more.

You’re exaggerating or you are an extreme exception.

Given the number of qualifiers (“successful senior programmers”, “private code”) I strongly suspect you’re just trying to find a reason to justify your opinion.

What I see in your comment is someone who has decided the answer, and refuses to consider alternatives.

They are fathers, amateur musicians, or like to restore old cars. Just to mention a few things. Could they spend 5-10 hours weekly writing code and hacking things together? Yes, but that would be a significant chunk of their spare time, so understandably they do not.

Graduates?

> no enthusiastic references, and you can’t find anyone who worked with them who will vouch for their skill

This will make your diversity even worse unless you're extremely careful with it.

I do think that coding tests have more of a role for new grads, though.

Additionally, a person who can adapt but has lower specific testing skill might actually be better at everything. As in your coach example, a slightly rusty to player from years past might be better than slightly better new guy with no track record. You're taking a gamble either way.

The lowest rung coding test is only good to filter out people who cannot code at all, and that if simple enough. They tend to be stupidly worded though instead.

Trying for anything tough tests for specific tricks, tooling and skills. Giving a longer time task is comprehensive but rules against people who are actually busy, presumably working, thus already kind of tested.

If every basketball team did that, every player would practice for that 2 minutes of play and you'd have no idea if they would handle an entire game.

The effect is the same for developers: interviewing developers practice for coding tests; memorize common interviewing patterns, quick solutions, and junior level algorithm stuff from their university days.

If you were interviewing a player, you'd probably look to their past experience and teams they played on, their positions on the court, and you'd check with their teammates.