Before Matasano, if you don't count clients interviewing us.

Who asks programming puzzles? Read interview postmortems. The answer appears to be "everyone". We are probably just talking past each other: asking someone how to sanitize an input or parse an HTML document during a face-to-face interview is a "programming puzzle".

It's hard to imagine that Facebook, Twitter, Dropbox, Google, Amazon, Apple, and all the rest of the companies that hire software developers, who have more or less the same type of interview, have gotten things wrong, yet have been so successful in hiring good people and building good products.

Perhaps the real problem is that those companies have taken all the people who perform well at whiteboard interviews, and what's left are people who struggle.

Still, I wouldn't say the system is flawed because it selects a certain type of people. Clearly those people do well.

It's not at all hard to imagine that the best known company in the industry, the one that spends the most on direct outreach to clients, whose compensation system more or less sets the standard for the industry, a company that is essentially rolling in money, is able to make a conventional recruiting process viable for itself.

Having said that, if you talk to Google people who are close to hiring, especially about the process that connects "interviewers" to "recruiters", I don't think you're going to hear consistently good things about how the system works in practice.

Googled used to do the puzzle thing and gather copious metrics on how it worked out- it turned out that it didn't.

http://qz.com/96206/google-admits-those-infamous-brainteaser...

Note the word programming. If a brainteaser puzzle has nothing to do with the job you'll be performing then of course it won't indicate real-world performance as well as one that's related to the opening at hand.

Yet, these large companies are always desperate to aqcuire, or 'acq-hire' smaller companies.

It's easy to imagine that smart people's thought processes are imprisoned by their own egos. It is an easy trap for a smart person to fall into.

Algorithmic testing has real practical implication if the role requires it, which a subset of the engineers at the aforementioned companies will use.

The other practical implication is standardization of the interview process, rooted around an academic background in computer science. It's easy to test whether someone knows a "fundamental" (whether it's actually fundamental to the job is another question) cs data structure or algorithm.

The idea is, interview for square pegs, since it is easier to measure the quality of that particular shape at scale. If there is a high-quality peg that happens not to be a square, it is a loss that the system is willing to accept, provided that a large enough % of the good candidates are squares.

Great points.

Though I believe algorithmic knowledge is a lot more practical than people give it credit for. I'm not talking about crazy graph searches, or implementing A* in ASM.

Are you going to be efficiently searching strings, or reordering arrays, full time? Probably not, but you will likely come across a time where you will need to think about the consequences of your design, even in a simple crud web-app. And that stuff is important.

> It's hard to imagine that Facebook, Twitter, Dropbox, Google, Amazon, Apple, and all the rest of the companies that hire software developers, who have more or less the same type of interview, have gotten things wrong, yet have been so successful in hiring good people and building good products.

It could be also cargo cult.

If you are a startup, how you will add a technical founder or a first technical hire? Just by his/her ability to write O(log n) searchs in a whiteboard or you will dig deeper and make sure is the perfect match?

Hopefully the startup will dig deeper.

Early in the life of your company you're hiring for very long term, and you need generalists.

Unless your company is building a product around whiteboard coding or something.

The point you are grazing is that this approach results in many false negatives, and few false positives. This is workable for companies that are consistently able to attract large numbers of candidates, such as the ones you listed. (And yes, there are false positives. I personally witnessed someone at a large company who was very adept at whiteboard coding, and a very lazy employee.)

You've said it more eloquently than I can, but I want to bring up that:

"results in many false negatives..." is not some universal truth.

Is there evidence that shows current software interviews turn down way too many great candidates?

There should exist a large enough group of developers who have been rejected a number of times[0], who have gone on to be very successful[1] in their careers. Does this data exist?

I agree that the current process isn't perfect, and it does produce some false negatives and some false positives. The amount which, has yet to deter me from beliving that the system works, as intended. Certainly there are some fringe candidates, who are turned away, but I don't think it's as onerous as people are making it out to be.

Also, most companies didn't start out with unlimited piles of cash, nor could they have their pick of employees. They had to work at it, and I don't believe the culture of their hiring has changed significantly.

For the record, I struggle with traditional coding puzzle interviews. But I also know coding on the whiteboard is a skill, which can be practiced, and with time, improved upon, such that if you are a really good developer, but struggle with passing interviews, you can deal with that problem by becoming better at interviewing.

Or you know, create something awesome, and never have to worry about interviewing again.

[0] Failing one or two interviews isn't really indicative of anything. So this group would require consistently failing multiple standard coding interviews.

[1] How to measure success? I say gone to be a major contributor to the success of a company, or built their own company that is considered "successful."

We turn down candidates all the time. No doubt that many of those go on to find success elsewhere, especially the ones that we turn down for fit (cultural or candidate expectations).

It's an open question whether a negative for fit is a false negative or not. If someone wants to be a foo and the need that I have is for a bar, it's a true negative for us, but when that persons goes and finds success as a foo somewhere else, I'm not sure where to put them your data set.

Even for the false negative, how would I possibly know what career path they took after that? I certainly hope it turned out well for all of them, but I have no practical way, not to mention no incentive, to find out.

To flip things around: would you hire someone who was a really good fit, but was weak technically?

I'd add to this that either way, whether interviews work or not, the problem is more complex than that: is the potential hire at the right point in his or her life to work for you? Are current open positions a good fit? Will the team get along? This may speak more about less mature candidates, or for smaller companies with more specific skill requirements, but my ultimate point is that rejection shouldn't be forever but few companies take that route...

Your post seems to employ circular logic. You're asking for evidence of one of your assumptions:

> Failing one or two interviews isn't really indicative of anything.

Apologies for the logic fail.

I'm asking for evidence that shows candidates who have routinely fail traditional software interviews, have gone on to be above_average -> exceptional in their careers.

My hypothesis is that if traditional software interviews have systemic and endemic flaws--that is interviews produce there are a lot of false negatives, there would exist a large group of people who have failed continuously, but have yet gone on to be good-to-great elsewhere.

Getting a no hire from one or two companies, would not classify a person as a "false negative", it could be (as others have mentioned) an issue with fit.

Well add 1 to the "rejected but went on to be awesome" Brian Acton.

I've had to do a bit of sanitizing input and parsing HTML documents "for real" at my job. It's not a primary duty, of course, but it comes up now and then. It seems like a reasonable test for "can they actually write some code to a well-defined spec," although probably not a great test for "are they good at working on a team and architecting bigger projects."

This is also a practice that is extremely easy to teach and fix.

But can be very costly to your business.

SQL Injection--not html parsing ;)