Here's my own example. I've been recording some of the problems I have to deal with at work, and some of the things I come across in my spare time in the form of a blog, for the last 3 years (with some gaps here and there). I recently started looking for a better place to work. I included the link to my blog in my resume. I got an offer recently, it will be a substantial increase in salary. I only interviewed with ~5 companies to get an offer, that's my easiest offer and shortest job search ever.

Now, would you say that I 'gamed the system' because I had a blog, and most don't? I guess you can say so, because when I started my blog, I had that in mind - it may be useful next time I look for a job. But I did not mindlessly type stuff in - I made sure I understood it as much as I can, and quite often writing the solution down helped my understanding. Same with this guy - he did not just memorize how to do it.

I had one YC company ask me to create a linked-list structure in ruby, and then reverse it. I did it recursively, mentioned that it would blow the call stack, and then re-implemented it iteratively. I also remember there was some problem where I used symbols rather than strings and that would cause some bad memory leaks over time in a long-running process. - From this passage, he might not be the "top 1 percent" everyone tries to hire, but at least top 5%, that I'm sure of. Or maybe he gamed my impression too.

That means asking these questions is akin to you taking an approach to judge an candidate on things which have nothing to do his job. That's why so many programmers don't about all this. The reason is for 99% of the work they are likely to do they don't need to know it.

Under such circumstances when you ask these sort of questions the only way candidate can deal with them is not by actually learning how to solve them. But learn them just for the interview sake.

So you end up hiring some one on a criteria doesn't matter. And the candidate manages to beat you by system of work which defeated your original means of judging him.

I think this is what gaming the system means.

Turns out it is pretty straight-forward in Erlang: lists:reverse(List).

Reading their contributions on github is a much better way to get a sense of someones capabilities.

Since most abstractions are leaky, you can make a good argument, that learning how libraries tick, even if you are never going to rewrite them, is a useful skill.

I don't think that list reversal is a good interview question. At least not as a positive discriminator. Though it might help you weed out people: I'd probably not want to hire people who can't figure out how to reverse a list. If they'll stumble upon such a simple problem, they might also stumble over the more complicated problems we have to solve to deliver business objectives.

I wrote `can't figure out' deliberately. Not knowing how to reverse a linked list, but being able to come up with an algorithm after a few minutes thought is perfectly adequate.

In particular I don't want to hire the people who would rather write their own implementation than use a library (think Not-Invented-Here and Yak Shaving).

I don't need to write Ruby in C.

Obviously I said yeah, I'd be willing to talk. They set up an in-person interview. Just 45 minutes they said.

So I show up, fully expecting a discussion about that area, my background, etc.

Instead, I get a random coder who wasn't even making eye contact when asking questions. And the questions? Elementary CS stuff that any undergrad would know.

Now, obviously I know them too; heck, I taught Algorithms courses in grad school.

But why waste my time with this crap? You can hire almost any fresh graduate from a top-40 school who would do a halfway decent job at answering those questions. What does asking a PhD "how to print the levels of a binary tree, one per line" tell you about his work? (That wasn't the question, but that was the level of questioning).

The sad part is: I can waste a couple of weeks of my life, go to Glassdoor etc. and basically memorize these questions. I'll probably stand a good chance of blowing the interviewer away by pretending to solve them on the spot. But what does that tell you about how I am at real-life problems?

Some people are probably bullshitters, but my theory is that some are just people who used to code (a little) but have since moved on to making architectural diagrams for so long they've forgotten what it's like in the trenches. Unfortunately, if you're interviewing for a job in the trenches, you need to demonstrate the ability to do the job.

It's entirely possible they screwed up, depending on why they asked you in, but once a company starts letting in people they "know" can code without verifying it first hand, things go bad fast.

I'm sensitive to that argument, but I also think the parent has a point. Coding the solution to a toy problem on a whiteboard has only limited correlation with the ability of a candidate to solve real-world problems. We use the tool because it's one of the best we have, not because it's a great tool. And if your interviewers can't summon the energy to make eye contact with a candidate...well, they're one-tool cavemen. Most software engineering demands communication skills and culture fit, yet we're still screening as if individual mental horsepower were the most important factor.

The danger in dismissing someone because they can't stand at a whiteboard and barf out the code to print the levels of a binary tree is that it has next to nothing to do with real work. That's just a fact. I probably couldn't do it correctly on a whiteboard, on my first try right now, and I write a lot of code. And I'm sure some hiring manager would eagerly dismiss me as "another frickin' PhD that doesn't know how to write code."

The point is twofold:

1) As in all things, good judgment is key. Sometimes the candidate can bork the technical question, and still deserve a hire.

2) If you find yourself depending exclusively upon the outcome of coding puzzles to screen candidates, you've already failed.

My approach is to throw out the whiteboard coding and logic questions whenever I can and to look for examples of a prospective employee's real world work, namely contributions to open source (e.g. GitHub makes this easier) and if not that, then perhaps they can directly provide me with private real world examples of code they've written.

I realise that not everyone will necessarily be able to provide such examples of real world work and yes, looking over someone's pre-existing project will probably take more time than seeing if they can reverse a string. But maybe this is also part of the problem: hiring practices in some places have gotten lazy.

I can't think of a way to entirely, in 100% of cases get rid of FizzBuzz, and the guy who has to escape the fire spreading from one side of the island etc. but I really wish I could, because it just doesn't feel right.

I've had some frustrating experiences where the obviously talented candidate was nearly passed over because they didn't use the right software engineering buzzwords while at the same time the manager came close to overruling the technical veto because they felt some other idiot sounded like a good fit.

Teaching someone to write unit tests is, imo, far more likely to succeed than teaching someone to understand recursion.

Most of the so-called algorithm experts I've seen fit into this knows-theory-but-no-practical-stuff category. The fact there are some list of questions and their standards variants. All you need to do is just crawl over interview forums. I assume they even have books, as in pdf ready made for these purposes. Just ensure you read all those questions before hand. May be spend an hour a day to get familiar with them.

Go to the interview, when the interviewer asks the question act as though you've never heard the question. Act like you've been having it tough. But then suddenly like a hero emerging from a crisis present the solution to the interviewer.

You have not clue how many people do this. I've seen candidates, whose only job is this. They change a company every year. Spend half their day everyday studying and collecting salary and interview trends. Apply to a big brand, game the interview collect the x% hike stay for an year and then move on.

These kind of people are an online club. And they tend to hire only their kind.

No regards for the guy's actual work knowledge, his ability to solve real world problems. His knowledge of a programming language, tools, techniques. His productivity all that is irrelevant.

All that matters in these large web companies is a Ivy league brand and theoretical knowledge and some arcane facts memorized at college. Close to 98% of the so called algorithm experts fit into this category.

The genuine 2% are at places they like to work at and surely they pick the place they want to work at and not the other way around.

I suppose, I just don't see why a shitty programmer couldn't be filtered out almost immediately through this technique. Maybe it's a company size/age difference - what size company do you have in mind?

(Also, at big companies, people cannot be fired. If you fire someone, their manager "loses an open" and they are less likely to get promoted. Remember, you have to deal with the bad employee, not your manager.)

That being said, I think a dismissive or patronizing attitude towards a coding interview would be mistaken. Allow me to explain.

It is quite well known that the ability to code is poorly correlated with holding an advanced degree or other formal pieces of evidence. A candidate with strong credentials understands this and should, therefore, not be in the least offended by being asked coding questions. As a good professional, the strong candidate would handle the questions with panache and should be ready for more difficult questions, discussing advanced topics linked to the question at hand, or discussing her particular area of expertise. Either way, she must be able to code her way out of a paper bag, and I see nothing wrong with the interview process probing for that.

I've interviewed a few dozen engineers while at Facebook, many with advanced degrees (because recruiters pair as well as they can interviewers with interviewees for areas of expertise), some from top 10 universities. I ask 2-4 coding questions extracted from my daily job, starting from undergraduate level and on rare occasions ending with a difficult complexity question. Nobody has ever done perfectly well, and for a variety of reasons holding an advanced degree doesn't correlate strongly with doing well.

All in all, I think you'd be mistaken to believe two weeks of rote memorization would be enough to pass the Facebook interview, which is very difficult. Overall, an attitude of professionalism, modesty, and focus helps a lot in any interview.

I almost always have a programming question I just made up 30 minutes before the interview. I make sure that I can easily do it in 10 minutes and then ask it. It really isn't important to me if they get they answer, but I want to see what questions they have, and how they think. Good candidates will usually get it coded, but some may get hung up on something early on, but still make good progress.

If they come in with a precanned response to my question then they're either a mindreader or can time travel. In either case, I still went them on my team.

This is a lot easier said than done. Do you really do this and, if so, any tips on how? (I'm guessing variations on a core set of questions.)

I usually don't mind asking a questions someone may have already heard because there's not really a right answer, but I just want to discuss a problem to see how you think through it.

Probably my only real tip I can think of is not to have them code something related to stuff you're currently working on. You'll be far more likely to underestimate how hard the problem is.

But the problems aren't that hard to think of. They're not like ACM programming contests. They're more like:

You have two lists. One is a list of IP addresses. The other list contains IP addresses or IP addresses with wildcards. E.g., 192.168.$.1 or $.255.255.255. Return the set of IP addresses from list 1 that are matched in list 2. Additionally return the index from list 2 that matched -- I want the index that is most specific (as defined by having the fewest wildcards, or wildcard furthest to the right when there is a tie).

If its a short list, who cares?

If you care about it all the time then you a premature optimiser - basically a hand-grenade with the pin out.

(by you I mean 'one' here - not you AdamT)

I don't think gaming the system is that trivial - an interview is a conversation, so things like computational complexity, runtime restrictions, data structures, implementation differences in various programming languages, and other things are bound to come into conversation. Now, if one can maintain that conversation as it moves along, and seem like a pleasant person to work with, it seems that they're no longer purely "faking".

Generally, the reason I ask such questions of candidates is to make sure I'm dealing with someone who can actually analyze a problem and get a reasonable result. So if they nail the questions because they took time to study and get good at interview-style brain teasers, that's actually fine with me - it proves the person can execute a plan to get correct results.

I'm not sure I'd call anyone who successfully "games" the system incompetent.

  - Do you know what SSL/TLS is?                       general
  - What's the maximum size of SSL protocol record?    specific
  - What's a cipher suite?                             general
  - How is DH-anon suite is set up?                    specific
  ...

  - What is a linked list?
  - What is offsetof (and how is it relevant)?
  ...

  - What is TCP?                                       general
  - What is TCP/Vegas?                                 specific
  ...

With the right set of questions, and some hints when the candidate struggles, you can very effectively gauge a candidate's limits on a subject, and see how they try to puzzle things out when they don't know an answer but know enough to take a good guess.

The key distinguishing factor of this method, IMO, is a candidate cannot memorize enough answers to bluff their way through a good questions list. If a candidate really is an N out of 10 with technology $FOO like they claim, they should have no problem answering most/all of your 50+ N/10-level questions for $FOO (my technical interview consisted of ~2 hours of this sort of interrogation for a straightforward junior web dev position. Interview length, in this interviewing style, is important for ensuring you've exhausted a candidate's book knowledge).

People will feel confident until they hit their limits. Their limits are what you are looking for (presumably), not their ability to think clearly when they are scared they just blew the interview on a question you didn't really expect them to answer.

For people hiring, go ahead and use this as a first-pass filter, but realize that it's easy to game (by someone who's trying hard and learning, so there is a positive signal in that). It's going to be important to go through and work with them for a period of time in a let's-date-before-we-marry relationship. As an employee, I always insisted on a contracting relationship before an employment relationship.

Like I said in the footer, Heroku really seems to do this the right way, and it shows. Kudos to them.

And everyone else needs to step up their game.

Both where that line is, and how they handle the failure tells you a lot.

(Caveat: Make sure you're keeping it on the fail/no-fail edge, and you treat the candidate with respect. The point is not to show them that they're wrong, but to explore the boundaries of their skill set. Without being mean is a major goal here)

And if you can't push the candidate there: HIRE THEM. Right away. If they know that much more than you do that you can't get them beyond their limits, they are a fantastic addition to your team. (Provided they match personality-wise, too)

As most of a job is about doing things that haven't been done before, getting the candidate into unknown territory is when you really find out how effective they will be. Some people just stop, others switch seamlessly into problem solving mode and come up with an answer that may or may not work, and also with a plan to test whether it does.

A candidate and interviewer both have to be fairly relaxed and comfortable to do it properly, but that's more of a problem for people new to the interviewing process (probably not senior management roles)

Some companies do this intentionally without hiring anybody in order to double-check their plans. It's basically free consulting.

If you're interviewing for programming jobs, the (low) calibre of existing applicants has probably already terrified you.

I think it is not necessary and not possible that the interviewer know more about every subtopic than every candidate. It is enough that they have comparable knowledge in most topics. Good candidates can very easily know more about some parts of the subject (and maybe less about other parts). Being a good interviewer is a really tricky business, because good interviewers should see the strength of candidates and should see if the candidate know even more than them about some subtopics. I've seen some bad interviewers who asked only the very narrow topics they knew well, and basically wanted exact copies of themselves.

On the other hand, there are plenty of competent programmers not willing to put up with this nonsense, and the companies are going to miss out on them.

There are people I know personally that have gone through every single one of the 1000+ Google interview writeups in glassdoor in order to prepare for interviews. One of my former co-workers showed me 10 pages of questions he distilled from glassdoor of various questions that were reported, and he memorized the answers to all of them, ranging from "find the common descendant of two nodes of a directed graph", to programming an AVL tree on the spot, to pure dynamic programming questions.

If they asked these types of questions 15+ years ago when I started, there's no way I could have gotten into programming. Even now, the caliber of those questions are simply too hard for me unless I sit down days before the interview to get the answer. There's no way I would be able to figure those out within 45 mins let alone 2 hrs, and I'm not dumb. It's ridiculous the level of expectations people have these days.

Sorry, `diverse' is an attribute of the distribution of questions you ask. Not of any single question. If for each interview you take a random question from glassdoor.com (and perhaps even tweak it a bit, if you feel like it), that might even meet the definition of a diverse but not too hard selection. (Unfortunately, I know almost nothing about glassdoor.com, and what kind of questions they have.)

I'm also part of a small start-up (as an employee), and I can certainly vouch for that. I for one put a very high emphasis on the personal/professional projects the guy/dudette in front of me has been involved in, on how best s/he can explain them to me, on how passionate the person seems when explaining all this (and no, I'm not talking about the enthusiastic approach that can be easily faked, I just think that when you're a programmer you can feel what the other programmer in front of you is very interested in).

Also, this mania a lot of interviewers have with the interviewees "always ticking the right boxes and giving the correct answers" sort of scares me. It's probably because I work in a start-up, but I can assure everyone that at least half of the time we ourselves don't know the answers (assuming that we even know the correct questions), and there are lots of times when we seem to be knowing in what direction we're going but truly speaking we don't.