I agree, but only if you're allowed to use references/google/etc and given a reasonable amount of time to accomplish it. I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd.

The idea that you would ever have to do something like this in a bomb-diffusing type of scenario with no internet and a ticking clock, would only test if you have already solved the problem before, not whether you could investigate a new problem.

Employers that do this are ridiculous. As are educators, too. What is this, preparation for when coders are kidnapped by terrorists and forced to recite how to tar/untar stuff on command? (relevant xkcd, of course)

In my experience, they're usually startups or smaller companies that are trying to "copy google" in their hiring practices, but don't fully think out how they're doing it. It's not a good sign.

:-D you never know!

Unless they are trying to hire a mentat.

Personally, I think it's as absurd as writing shell scripts on a computer with no `man`, or without being able to refresh your memory with `<command> --help`. Good luck with that!

I was in an interview where they asked me to configure specific firewall rules on a notepad with no access to a terminal, so I couldn't check man iptables for stuff like how to match certain sources/destinations hosts/IPs. And I was not too familiar with the opts (was it --dest, --dst, or -d? and how do calculate the network mask for this range of IPs?). It was also for a junior position so I thought they were taking a piss at me and simply told them so.

The candidate was kind enough to avoid posting the name of the company (to hopefully hide the walk-through from Google queries for "crack me hiring test company name", but an HN discussion might then arise around the walk-through, in which the company would be named, and well, there goes your test.

It's hard to accurately run "take home" tests when hiring because of just this.

You're asking a potential candidate to spend hours cracking your program but you can't be bothered to change it for each batch?

I expect any company applying this interview technique to at least put some effort into it and even then it pays off since a batch could contain hundreds of candidates and you only have to build the program once.

I strongly suspected that sometimes goals of the person responsible for hiring are not quite aligned with the company they work for. Example:

Company's goal: Hire an exceptional candidate.

Lazy or overworked hiring manager's goal: Reduce the size of the stack of 1000's of resume's I have to go through using any means possible that will make it appear like he/she actually did the work to find the candidate.

"Lemme see, use keyword search to ditch the ones without 'Github' on their resume'. Now ditch the ones who don't have a college degree. Still some resume's left? OK, how about tossing the ones without a CS degree?"

Run a filter with enough conditions and you can get that pile down to nothing pretty fast. If that is still not enough filtering, ask the candidates to do something very time consuming. This a great way to weed out people who are very busy or exceptional at what they do and don't need to jump through your hoops.

I'm surprised they didn't make him sign an NDA for it...

Make it slightly wrong, or obviously fingerprinted.

>>I've had a "real-life" test where I wasn't allowed to verify or look up information, or where I'm giving a very short time to execute, and I've always thought those were absurd.

You certainly haven't worked at major IT services firms here in India. I spent early part of working years (around 2006) coding by purely reading books and discussing things with folks on internal mailing list. Because only managers and levels above were allowed internet access.

>>The idea that you would ever have to do something like this in a bomb-diffusing type of scenario with no internet and a ticking clock, would only test if you have already solved the problem before, not whether you could investigate a new problem.

This is unfortunately true even for coding interviews. Most algorithm questions are impossible to answer unless you have rote memorized the algorithms and their implementations.

By complex, I mean the question should be difficult enough that you can't easily hold every parameter, case and step in your head: I want to see how you deal with situations that you have to mentally walk through and issues that you might lose track of.

By potentially elegant, I mean both brute-force and optimal solutions should be able to fit on a small whiteboard, without requiring more than 2-4 variable declarations and no more than a nested for loop (or some sort of recursion). The optimal solution shouldn't require putting together pieces that they aren't really expected to know/use, i.e. no more complex than arrays, maps, strings and ints, and any other "data structures" the candidate ought to know, given their background. For instance, if they claim to be a crypto expert, they should know how to use a black box that gives them private and public keys.

By straightforward I mean, you don't want the candidate to have to have an "aha" moment. If you're asking for them to write a max-flow variant, you'd better be explicit that you don't mind a really brute-force solution, or most of the interview will be spent trying to remember complex algorithms. Is the algorithm one that you can understand by walking through test cases/figuring out what steps you took in your mind to go from input to output? If it takes more than that, you're going to get a few false positives, i.e. "Wow she was great! No one's answered this so fast!" (because she just spent 20 hours doing this on a homework), and many false negatives.

Of course, there's tests like user:B5Geek's, where the candidate gets a PC with a slightly-unplugged network cable and has to realise this fact.

In a lot of ways, it's a matter of identity crisis for software developers. We're all, industry included, not quite sure whether we're mechanics, carpenters, architects or scientists.

I know I am far less tolerant of bullshit interviews than I once was, and would have far fewer qualms vocalizing what I'm thinking - "I don't know. If I saw this in real life I'd check Google, because it reads exactly like the sort of contrived problem used in interviews and thus is well documented, and likely solved quickly by those who have memorized it. I am not one of those. I'll take a stab at it now if you just want to see how I reason through it, but if coming up with an efficient, correct implementation is part of your criteria we can probably save both of us the trouble" - is that considered acting like an ass?

Surely there's more efficient ways of evaluating that.

I confess I don't know how I'd apply this to developers. They pass fizzbuzz, OK - what do you do next?

I know what I would do, just have a 10 minute conversation with them about technology. I can easily figure out just where a person is in regards to their development as a programmer just by listening to them talk about what they're paying attention to in the tech landscape. If they don't blink when I mention Hacker News, I know they're on the right track. Do they know what Go is, or Haskell? Do they have an opinion about Node.js?

These criteria are intended to gauge motivation and enthusiasm in the absence of an impressive track record. If they're capable but not highly motivated, then that's fine so long as you're only hiring them to do stuff inside their comfort zone. We have a Microsoft guy just like that. If they're motivated but not capable, then with coaching they'll be able to do anything. The best way to discover someone's motivation is to get a gander at what they're thinking about when they aren't working.

If you're looking for someone who is motivated and capable, you can eventually find someone like that, but you'll have a hell of a time keeping them. Developers have so many options these days that you need to have at least 3 of: competitive salary, decent perks, fun work environment, a glamorous field; in order to hold on to them for longer than a year.

Most companies are lucky to manage one, if they're very lucky two. Since salary is often the easiest thing to manipulate and honestly the most important to the employee, err on the side of paying too much. It's expensive but not having competent developers is far more expensive.

Programming is a wide field, there's much more to it than the front page discussions on HN can cover.

Of course, if you're looking for a Web developer to make "cool new stuff" for the latest fadgets..

The ability to solve common problems acts as a good proxy for experience, If they can't google how to fix a retain cycle or a broken constraint, we know they probably don't know what they're talking about.

The ability to build a good, small system on top of the broken app acts as a good proxy for development talent. If they can't easily set up a dead-simple delegate to do an asynchronous web call, they can't hack as a mobile developer.

Most importantly, if they don't care enough about the job to take two hours to whip up a super-simple app, then we don't have to waste our time interviewing them.

For example, an IBEW apprentice will do a lot of gruntwork on the job site. Pulling wire, for instance. But they'll get exposed to how things are done, to trade industry expectations, and what a project looks like from start to end. Part of the week, they spend in a classroom learning the specifics of their trade. Not just the high level theory, but the nitty gritty, based on real world applications and problems. This apprenticeship lasts from 2-5 years depending.

I don't see why this model wouldn't work really well with programming.

You don't get constant mentoring at the company, but can ask one of the developers or fellow apprentices when you're stuck with one of the tasks you get assigned. Or for example, any time a developer had a small task that he felt, I couldn't do by myself, or it was too time critical he just asked me to sit beside him and watch, listen and take notes while he did the work. How it works is different in any company but the outcome should be the same: real world experience on real projects with real clients and their time/budget constraints.

That means, once your finished and a professional yourself, it isn't a question of interest but a completely normal thing to help out the apprentices. Because that's how you got to the point where you are.

We may laugh, but starting in technical support gives you an idea of how your users are using your product, if your documentation is accurate and helpful, what the pain points for people not in the department you are destined for are, and gives you an idea of how the product works.

It also quickly weeds out the people who can't effectively communicate before they get to the programming team.

I have only seen it (co-op) used in the last few years.

http://www.ceiainc.org/history

http://en.wikipedia.org/wiki/Cooperative_education

Algorithm tests only measure the number of man hours the candidates spend on career cup daily. Or how good they are rote memorizing things.

There are many ways to check how good a candidate is at doing their actual work. For instance you can take a candidate to a code review and see what kind of inputs they contribute to the discussion. This alone will give you good deal of idea on how well they understand code and quality. There are other ways, like say asking them to write unit test cases for a class- That would give a lot of idea on how they think with regards to breaking and fixing things.

One more way I have after doing basic checks is pair programming, or picking a totally new problem and working with the candidate to see how they think, how they work and how good their communication skills are.

There are many other ways too. Recently I've been given coding assignments for very practical applications, though those turn out to be a little difficult and time consuming. And at times the expectations are very high. Like asking the candidate to complete a whole application in very short times. It can be hectic with your current job, and a family. But generally those are a good indication of how good the candidate will do at their actual work.

Have a computer setup and running (all properly configured). Pull the network cable out of the jack a little bit (so it looks like it's plugged in but isn't). Ask the person being interviewed to show me an IP being used by microsoft or google. (so ping/dig/nslookup/etc)

Let the person know that (a) the computer is in working condition (i.e. no drivers are missing) (b) the network works (i.e cables are good, switch is good, DHCP is enabled, etc.) (c) tell them that this is a test to determine their troubleshooting skills

It is always disappointing to see how few ever open a term/cmd window (depending on the OS). 90% of participants just try to open a web-browser and type in "what is the IP address of google"

So you lie to them? I don't understand what the point of this test is. My first inclination is to open up a term and ping google but I would be pretty annoyed that your "first-wave" test involved actively lying to a candidate.

Yes, I agree. Are we supposed to also assume that our bosses and interviewers lie to us, too? At no point does he say that he is emulating a user that needs support help.

It's just testing the basics really

This means - to me - that the network... well, works. And it clearly does not. If this was a Windows machine I'd probably wonder what the hell was going on with the Ethernet (!) sign in the bottom right of the tray and if it was a decent version of Ubuntu (11.x or earlier) then you'd also notice it quickly.

But still. He's pretty obviously leading the person astray by giving him the solid impression that the machine is in good working order. (which it is not) If you mean to say that he didn't CLEARLY specify that the "cable was fully seated..." I don't know what to tell you. That's a "gotcha" question. Not a good diagnostic one. Unless this is for a $12/hour CSR position. In which case, OK fine.

If you can't solve a problem with only some information, and some of it incorrect, then how are you supposed to be able to help?

Oh god.

  Windows key + R / "cmd" / Return / "nslookup google.com" then when that fails, "ping 8.8.8.8" and when that fails, "Ipconfig /ALL", then ask "Do you use Dhcp or are ip's assigned?" as I reboot the computer and check the physical connections at the same time.

"ping goo.gl"

http://74.125.239.9

    win+R, ping -t google.com, return

A good hacker is a lazy hacker.

You should always remember that hiring is as much about persuading the best person to say yes to a job offer as it is about weeding out the bad people.

Job please 8-)

If you are asserting that google is not using the IP of it's public facing DNS service, this company is full of horrible tech people, so nevermind: I don't want the job.

Sloppy work Google.

Are you really asserting that a google owned, and google operated service is not using the IP with which I access it?

Easy question

I believe there is a happy synergy here, by the way. A candidate who welcomes this type of test -- who sees a direct work sample as the easy and efficient way to prove themselves to you -- is probably someone who knows their work is good.

And if your "technical grilling" fails to identify strong candidates, you're obviously asking the wrong questions or, at the very least, not asking the right ones.

Your objection is a fair one, but having come into the company through the process I describe I am fond of how it let me (a) show that I could actually do the stuff I claimed to be able to (b) show my thought processes and how I would deal with having received a box in this condition. So I was quite pleased to do it as a candidate. Certainly less of a waste of time than a clashing interview.

The way I do this is the real coding test is only administered to final candidates. That's a bit kinder since only a few people with a serious shot at the position end up having to commit the extra time and since we observe the whole process (and discuss) it doesn't waste tons of our time either. Whiteboard and discussion can reveal a lot but IMO there's nothing so revealing as actually watching someone work.

As an aside I would much rather be interviewed this way than at a whiteboard. At least for me there's a certain amount of my programming skill that's nonverbal and flows easier for me at a keyboard.

A cook is someone who can be hired with very little experience. A chef is someone who is trained professionally. And plenty of chefs are hired without a "taste test" based on their academic and employment history.

They're hired on a provisional basis, true, but they are paid while they prove themselves.

So in the end you're only hurting yourself because you'll mostly get naive and easily manipulated people who are fine for the lower ranks but absolute poison once they move up. That's if they don't wise up and leave first, costing you time and effort to find and train replacements and scramble to rediscover and distribute the lost knowledge they left with.

But then again, such a long duration challenge is already very telling of the company culture. If you can't respect other peoples' time, then your company is already on shaky footing.

These things shouldn't be designed to take 14 hours. They should be designed to take one, maybe two or three . . . if you know what you're doing.

One of the other posters took so long because he didn't know what he was doing. Well, perhaps that is an uncharitable way to say it, but you know what I mean. :)

Yeah but how many questions must you ask before you are certain? Otherwise you're rolling the dice that the candidate has simply seen that problem before but has terrible habits otherwise.

Reduce the time required and you risk more false positives. Increase the time required, and you risk the skilled ignoring you.

Given how fast things change, even if you imagine the person staying for only a year or three, learning abilities are nontrivial.

I am going to go ahead and challenge the assumption that your willingness and eagerness to learn on the job is not really correlated with how you spend your time outside your working hours.

As an interviewer I've done something like this when needing to hire for a very straightforward job with very little time and an effectively endless supply of candidates.

This addressed the problem of our time crunch, but I doubt it helped the selection process overall.

I think this sort of testing is subject to all the same issues as most any other method of interviewing, but limited to a much smaller sample size.

Basically, it's arguably good for the interviewers and a coin flip for everyone else.

As an interviewee, I haven't encountered a test like this, but I expect my response would almost surely involve questioning the architecture decisions that led to such a problem in the first place.

In this case, as a sysadmin I was pleased to be able to show my chops in a realistic test. And our previous hire was also quite pleased to get a real test. But of course, there is potential to abuse it.

I know many so called sysadmins who could not pass this test in a reasonable time. Sometimes people without even basic script-fu or networking knowledge are considered sysadmins. And this drives me crazy.

"devops" is fighting words in our team. Damn, if I could get our devs who'd like "devops" on their CV to give a hoot about the "-ops" half of that buzzword ... when a dev shows awareness of ops issues with their shiny new thing, I make a point of mentioning their name positively to the dev manager. "X knows their stuff. Give them more good stuff."

Always the ops guys are in fault.

I think every developer should be given sysadmin 101 and 102 lessons.

I'm not sure how to educate devs to think ops. Perhaps have their phone ring at 3am when stuff breaks? I think that would close the feedback loop nicely.

(Let me note again: I LOVE the devs who think operational issues, who think end-user issues - what customers are actually like - who think "developer of the whole thing from go to whoa" and not just "coder on my PC." As I'd hope we all do here.)

Sometimes you just need to hire more people or ask the rest of the team to accommodate someone that's that one random oddball. A management book I read mentioned Phil Knight talking about how the Bulls had room for one Dennis Rodman and only one, and made it clear that nobody else can pull the stunts he does without disrupting the harmony.

There's a huge difference between a dev that does something a bit out of ignorance and one out of indignance.

Some companies have developers spend some time in SRE (I believe Google practices this sometimes) so they can gain some insight, but it may not be the best idea for a lot of orgs. It's part of why most orgs that do some form of devops well tend to remove a lot of concerns off the table by using stuff like AWS. Meanwhile, silozation helps people maintain some sanity and focus in larger orgs where there's so much BS work on top of your technical duties.

Sometimes the culture is short-sighted and people are at odds with goals though. I've been penalized by managers and peers for not paying enough attention to my dev duties (which were pretty meh) when I was busy helping support and sales help bring in and retain $2 million in accounts that they later named me on calls as their informal engineering MVP.

But really, being aware of what other people care about in their job is a contentious issue that I mostly think boils down to personality and general ideas of teamwork.

I don't remember how many times I had to debug instead of them or design better for them.

By the way, hacker news developer quality is really better than average or even better.

(Ahhh, the BOFH stories ...)

That might work on college students.

To me that screams that you can't properly interview your candidates.

> I note we've just done a round of interviews where we get a nice-looking CV and conduct a technical grilling.

You need a better screening process and a better layered.