The trick is, to work in a recruiting context, you also want those tests to be standardized and repeatable. A lot of companies fall down on this. They have candidates do "real work", often in a pair-programming context. There a bunch of problems with this:
(1) "Real work" usually isn't standardizable, so you can't compare candidates
(2) Signal quality from the test is intensely dependent on who is doing the work with the candidate
(3) Two different candidates might end up getting "tests" that are wildly different in terms of predictive power
I have a bunch of ideas for pure software development work-sample tests, but I'm not ready to share them. The idea is simple, though:
* It's a realistic exercise that approximates actual day-to-day work as much as possible
* Every candidate gets the same exercises
* The exercises have objective (preferably gradable) outcomes
I want to keep my hands on a keyboard and I don't like airplanes (which rules out consulting) so I've applied at a lot of companies to be a ``security engineer''.
I can't tell you how many well-respected companies ask me to write min-heaps, depth-first searches, etc. I don't understand what they are asking me this for...It isn't even close to a realistic representation of what my day-to-day responsibilities would be. It is also an immediate turn-off....
Cramming these questions online does not work. I don't know what Facebook/Google et al are doing but I can spot a candidate who studied interview questions from a mile away. They may bang out a DFS by rote but that's just warmup; they should be able to talk about details, deal with changing requirements, discuss the algorithm/efficiency/design of their solution, be able to talk about underlying data structure primitives, etc.
To be fair the grandparent was talking about a specialized position in security engineering for which the generic coding interviews could be a bad match.
And I don't understand why someone who spends his time up to the eyes in crypto proofs, exploit code, and research articles should have to cram for an Algorithms 1 exam for each interview.
I've seen these questions used well (had a neat discussion with a start-up CEO in Boston once about reimplementing a toy Twitter in Scala), and really brilliantly (another start-up in Boston once asked me to describe my favorite algorithm and got a short, ad-hoc tutorial in lottery scheduling), and really, really badly (BigCos asking stuff you'd see on an undergrad-level exam from years and years ago).
Strangely enough, I'm praising the two startups that didn't offer me a job (though I'd really like to reapply to that second one), and condemning the BigCos (you know who I mean) that did offer me a job.
I'm not saying it isn't important to know, but I think there are better indicators for someone who will perform well on the job
It's no surprise these companies rejects many really good hackers (at least from people I know... yes, I admit the small sample size) and accepts these just optimize for algorithm preparations by doing a set of online coding problems.
I wouldn't ask a candidate to implement a min-heap (mostly because it's kind of obscure and I'd have to think about it) but a depth (or breadth) first search is something I'd expect any competent candidate to rattle off in a heartbeat, just because it's such a fundamental cs principle that gets used in lots of places and, unlike things like sorts, doesn't really have a good library to fall back on in practice.
In reality, I have not accepted an offer from any companies who ask me these questions because most of the positions are not exactly what I am looking for.
Also, I tend to disagree. Some of the best security guys out there never went to college. It's hard to expect these guys to know algorithms and datastructures even if they are really gifted exploit developers, or the like.
edit: To clarify. What this really shows is that I am not being interviewed by a security person...ie there is no special interview process for my position compared to a regular software engineer.
My cousin's employer has started using Javascript Under Pressure [1] as the first round of technical interviews - it lightens the mood, and fully half of the people applying for a front-end development position at his company can't finish the first one, which is literally to write a function that returns the argument multiplied by two.
However, getting one of the tests wrong in 4th problem because of two chinese UTF-8 symbol having more length than "tiny" kind of irked me and reminded me why I do not do webdev.
I suppose this is actually a good kind of test, because real webdevs would actually have learned not to use s.length but use something else.
What is it though? This seems like a Javascript fail more than anything else.
Gah, that made me jump.
That would still be a notable effort, but I'd like to know who I'm hiring and for which qualifications (cat herding or design and implementation).
And would that matter?
When the teleprompter reading person applies for a security job they're supposed to do alone? yes.
Interview Process is: HR filter -> Manager interview -> Grandfather (Manager's Mgr) interview -> Peer meeting
The peer meeting isn't conducted in an interview setting. It's more informal, adapted to the needs of the applicant. It could be a lunch or a dinner and once it was even participating together at a hackathon.
We carefully select peers against "buddy bias" and encourage diversity in the workplace.
Our company is over a period of several years very successful in recruiting and have very little turnover. The biggest drawback could be that we are "too picky" and thus (at times) have problems growing fast enough.
The secondary reason to ask these easy basic dev questions is an attitude check. Someone who gets personally offended at being asked to do something that is easy for them has a risk factor for being a poor teammate.
I see statements like this thrown around a lot. Do you have precision/recall/F-measure numbers to back this up?
Depends on your language. Some are better able to abstract and express this `pattern' as a library.
We have a front-end and back-end test. I'm sure that at some point we'll grow large enough that solutions to our test will be posted online but I'm not going to worry about it right now.
"but only because a) he was really hungry at the time b) he really went out of his way to build up a lot of contacts with a lot of top-notch talent."
And that was really the essence of what I was going to say. I think after a while anyone who is busy and being paid in the manner that a recruiter (or real estate salesman) is is going to go for the easy route and the low hanging fruit. Especially as they get busier. Because back and forth is time consuming.
It's kind of like a satisficing model. You need to keep the client happy enough to keep sending you projects and no more than that. [1]
Consequently in theory a new and hungry recruiter is really the one that will give you the best results but that assumes they have a file full of candidates and that kind of contradicts being new.
[1] There is truth to that book that stated that real estate salespeople get more when they sell their own house than your house. 3% of $10,000 is not $10,000 and it pays to move on to the next deal in your book.
I've never seen a industry that needs a fresh, iron clad Application, but it has to be top notch. A hungry, smart developer could revolutionize the Realeste market with the right code, and eliminate Realtors to a history wiki. Plus--I would never need to look at their stupid smiling faces everywhere. And yes, I know a house is the biggest thing you will ever own--you need hand holding? That's all your full commission will get. Smart Realtors always cover their asses legally if you buy a Lemmon. This is just my rant on Realtor's--especially full commission brokers. The above post reminded me of how arcane buying a house is.
The large bodyshops, OTOH, seem like a colossal waste of time to me. I don't trust them, and the candidates they offer have been almost uniformly disappointing.
His "secret" is that process, which also allows him higher than average access to passive job seekers. Let's face it, many (not all, but many) of the "best" candidates already have a sweet gig and aren't pounding the pavement looking and mindlessly applying to every job posting. This recruiter calls them and they take the call, because he's not just matching asses and seats like it was a musical chairs problem.
Candidates like him (he placed me once 17 years ago) because he's not perceived to be a time-waster, and hiring managers like him because I get better candidates through him (albeit at a lower volume and I "can't afford" him for every role).
That's responsive to your first sentence; to your second sentence, I generally agree. I have the luxury of having a strong in-house recruiting team, who are willing to selectively and smartly use outside agencies when it makes sense. As a result, I get to direct the flow of nonsense from outside agencies at my in-house team. ;)
I treat recruiters like widening the funnel, and haven't found any correlation that suggests they produce better results.
I have found the best results is hitting people up through linkedin.
When was the last time you applied for a software developer job?
Who asks programming puzzles? Read interview postmortems. The answer appears to be "everyone". We are probably just talking past each other: asking someone how to sanitize an input or parse an HTML document during a face-to-face interview is a "programming puzzle".
Perhaps the real problem is that those companies have taken all the people who perform well at whiteboard interviews, and what's left are people who struggle.
Still, I wouldn't say the system is flawed because it selects a certain type of people. Clearly those people do well.
Having said that, if you talk to Google people who are close to hiring, especially about the process that connects "interviewers" to "recruiters", I don't think you're going to hear consistently good things about how the system works in practice.
http://qz.com/96206/google-admits-those-infamous-brainteaser...
Algorithmic testing has real practical implication if the role requires it, which a subset of the engineers at the aforementioned companies will use.
The other practical implication is standardization of the interview process, rooted around an academic background in computer science. It's easy to test whether someone knows a "fundamental" (whether it's actually fundamental to the job is another question) cs data structure or algorithm.
The idea is, interview for square pegs, since it is easier to measure the quality of that particular shape at scale. If there is a high-quality peg that happens not to be a square, it is a loss that the system is willing to accept, provided that a large enough % of the good candidates are squares.
Though I believe algorithmic knowledge is a lot more practical than people give it credit for. I'm not talking about crazy graph searches, or implementing A* in ASM.
Are you going to be efficiently searching strings, or reordering arrays, full time? Probably not, but you will likely come across a time where you will need to think about the consequences of your design, even in a simple crud web-app. And that stuff is important.
It could be also cargo cult.
If you are a startup, how you will add a technical founder or a first technical hire? Just by his/her ability to write O(log n) searchs in a whiteboard or you will dig deeper and make sure is the perfect match?
Early in the life of your company you're hiring for very long term, and you need generalists.
Unless your company is building a product around whiteboard coding or something.
"results in many false negatives..." is not some universal truth.
Is there evidence that shows current software interviews turn down way too many great candidates?
There should exist a large enough group of developers who have been rejected a number of times[0], who have gone on to be very successful[1] in their careers. Does this data exist?
I agree that the current process isn't perfect, and it does produce some false negatives and some false positives. The amount which, has yet to deter me from beliving that the system works, as intended. Certainly there are some fringe candidates, who are turned away, but I don't think it's as onerous as people are making it out to be.
Also, most companies didn't start out with unlimited piles of cash, nor could they have their pick of employees. They had to work at it, and I don't believe the culture of their hiring has changed significantly.
For the record, I struggle with traditional coding puzzle interviews. But I also know coding on the whiteboard is a skill, which can be practiced, and with time, improved upon, such that if you are a really good developer, but struggle with passing interviews, you can deal with that problem by becoming better at interviewing.
Or you know, create something awesome, and never have to worry about interviewing again.
[0] Failing one or two interviews isn't really indicative of anything. So this group would require consistently failing multiple standard coding interviews.
[1] How to measure success? I say gone to be a major contributor to the success of a company, or built their own company that is considered "successful."
It's an open question whether a negative for fit is a false negative or not. If someone wants to be a foo and the need that I have is for a bar, it's a true negative for us, but when that persons goes and finds success as a foo somewhere else, I'm not sure where to put them your data set.
Even for the false negative, how would I possibly know what career path they took after that? I certainly hope it turned out well for all of them, but I have no practical way, not to mention no incentive, to find out.
> Failing one or two interviews isn't really indicative of anything.
I'm asking for evidence that shows candidates who have routinely fail traditional software interviews, have gone on to be above_average -> exceptional in their careers.
My hypothesis is that if traditional software interviews have systemic and endemic flaws--that is interviews produce there are a lot of false negatives, there would exist a large group of people who have failed continuously, but have yet gone on to be good-to-great elsewhere.
Getting a no hire from one or two companies, would not classify a person as a "false negative", it could be (as others have mentioned) an issue with fit.
SQL Injection--not html parsing ;)
What does it mean? Throw engineering problems your CURRENT team has solved in the past at new candidates. I have seen only ONE startup ( even among the nastiest of hard problem seekers ) GET how substantial this is. So many of these companies are too busy to think about smart sourcing, at their own peril.
Test new engineers off past solved problems. End of story.
Can you speak to that a little? I'm reading it as either programming challenges in a real environment, or analysis of previous code snippets submitted. In either case, I'm happy to see more companies doing this.
I'm always looking for ways to improve my interview skills, but I want to do it honestly. Studying to the test isn't fun for me, I would rather hack on something.