News for hackers. Stuff that matters.
Re:How Google Hires by tptacek on Tuesday April 07, 2015 @02:28PM (#9336017)
I've written a bit about this: http://sockpuppet.org/blog/2015/03/06/the-hiring-post/

The worst thing you can do is unstructured interviews. Unfortunately, that's exactly what almost every company does: they create ad-hoc committees of workers and have them each design an interview, often on the fly, for each candidate.

Everything you do to add structure and remove degrees of freedom from interviewers will improve outcomes. Every company that hires more than one tech worker in a year should be working on this problem, and hopefully documenting their result.

Our structured interview process differed from Google in that we (a) completely standardized interviews and (b) designed the questions in those interviews to minimize the need for free-form follow-up questions ("Tell me about a time..." and then digging into the answer to mine conclusions). We tried to flip the script: instead of having the interviewer ask followups of the candidate, we put candidates in a position to ask lots of questions of the interviewer, within a framework that generated lists of "facts" that we could record and compare to other candidates.

Example: "I am going to describe a trading system in broad strokes. I know lots of things about it works, and I want you to ask me enough questions so that you can roughly diagram it on a whiteboard, capturing its components." Then, later, "I want to have a conversation where we rank these components in order of sensitivity to latency". The interviewer captures the answers, but the back-and-forth is dominated and mostly directed by the candidate.

The problem I see with the Google strategy is that it combines interviewer-directed free-form questions with an open-ended and somewhat fuzzy question that really evaluates less the candidate's ability to do the work and more their presence of mind during the hostile, stressful interview process.

Things I positively responded to in this article:

* The notion that we come to snap judgements about candidates within minutes of the interview and are then hostage to confirmation bias for the rest of the interview

* The effectiveness of combinations of evaluation techniques; for us, a combination of work-sample testing and scripted interviews were extremely effective.

Re:How Google Hires by hoboon on Tuesday April 07, 2015 @06:34PM (#9337630)
I wonder if it's common or I am just unlucky or unskilled...

I've been out of work for 11 months. I've been on a large number of interviews and it seems like after that many failed interviews, the answer is probably with me.

The most frustrating thing is the lack of consistent information about why I didn't get hired. Some people said I had good theoretical grasp but can't code. Others said I could code really well but didn't have the grasp of fundamentals they wanted. Most stay silent.

Frustrating.

I've grown to hate and resent myself.

Re:How Google Hires by shawndumas on Tuesday April 07, 2015 @08:28PM (#9338068)
I've conducted a lot of interviews. In my 23 yr career I've done on avg 10 a year. Early last year I was in Yahoo's hiring war-room and did more than I care to remember.

All that to say; I'd love to conduct a few mock interviews with you. My contact info is in my profile.

Do it. Contact me. Seriously.

Edit: this offer extends to anyone that wants help. I am a front-end engineer so that means you'll get more mileage out of me for JS, CSS, and HTML; but I am totally willing to help with the subjective side to interviewing as well.

Re:How Google Hires by steffan on Tuesday April 07, 2015 @09:06PM (#9338212)
Thank you for doing this. You've improved my impression of humanity today, and inspired me to do the same.

(And you also, papercruncher)

Re:How Google Hires by nashadelic on Friday April 17, 2015 @09:41AM (#9394194)
As someone who does 8 interviews a day and loves to help candidates understand what they did wrong and how they can improve; this is really, really nice of you.
Re:How Google Hires by philsnow on Tuesday April 07, 2015 @10:09PM (#9338437)
Did you mean "on avg 10 a [month]" ? 230 over a 23-year career doesn't seem like all that many.
Re:How Google Hires by grrowl on Tuesday April 07, 2015 @11:35PM (#9338660)
Maybe not heaps, but consider you won't hire at entry-level, and unless you beeline straight into management you're unlikely to be conducting anywhere near 10 a month, ever.
Re:How Google Hires by philsnow on Wednesday April 08, 2015 @12:37PM (#9341985)
That doesn't agree with my (somewhat limited) experience at all. I don't know if my experience is atypical or yours is, though.

I interviewed more people than that per year while I worked at Google, and I'm doing even more now that I'm at a smaller company.

Re:How Google Hires by shawndumas on Wednesday April 08, 2015 @02:19PM (#9342722)
As a software engineer you are doing more than two interviews a week on avg?
Re:How Google Hires by grrowl on Wednesday April 08, 2015 @07:41PM (#9344921)
I've worked at small places, where we might only hire 5 people a year; and larger places which were large enough to leave it to the leads (engineer-managers overseeing 20 working engineers)
Re:How Google Hires by papercruncher on Tuesday April 07, 2015 @06:58PM (#9337739)
Don't hate yourself, interviewing is a skill that can be improved with practice. I'm not hiring right now but if you're in SF, I'd be more than willing to give you a mock interview and all the honest feedback you want. Email is in my profile.
Re:How Google Hires by maheroku on Tuesday April 07, 2015 @07:12PM (#9337786)
I think the best way to get over that is to make something on your own that is so great that everyone wants to hire you without even doing an interview!

The funny thing is that you may not even need a job once you do something great on your own :)

If you are short of money, you can always get a simple part time job to survive.

Re:How Google Hires by danjayh on Wednesday April 08, 2015 @12:22AM (#9338772)
Last time I went out interviewing, I whipped up a fun little Android game, put it on my phone, and made a point of finding a way to slip it into the conversation. It was great because:

1) It let me set the tone and path for much of the interview, since so many places do unstructured interviews. 2) It was a concrete demonstration of my skills. 3) Java and mobile are both hot technologies, and the game also made effective use of other common stuff like Open GL, multithreading, network, database storage (via sqlite), etc. 4) Spending a few minutes playing a game sets a relaxed tone for the entire interview, which makes things easier for everyone involved.

Structured interviews really are a brilliant strategy. Whenever I interview, I do my best to subtly direct the interview in a way that exposes my strengths and leads the conversation into areas that I am most comfortable with (and trust me, whipping out a concrete example full of technologies that you're absolutely comfortable with helps). A structured interview, to an extent, would allow the employer to retain better control of the interview (whether or not the realize it), which is probably to their benefit. For instance, I teach Java, C#/VB, and some other modern languages at a local college after work, do tons of C / embedded / network / etc. stuff at work, and do digital electronics for hobby ... so if you let me push the interview in those technical directions, I'm at an advantage.

Re:How Google Hires by megablast on Wednesday April 08, 2015 @12:41AM (#9338817)
This is great.

I am always surprised when I meet mobile developers who have never created a mobile app for themselves.

Re:How Google Hires by danjayh on Thursday April 09, 2015 @12:09AM (#9345745)
The game is called 'aqua balls' and is on google play. I haven't updated it in almost 3 years, though ... right after I finished it I started having trouble with my wrists and finger joints, which meant no more coding for fun (I took a team lead job and cut out a lot of typing at work too). My brother wrote and maintains the other app published by 'woggle' (he also did all of the art for aqua balls), so it is still updated regularly. He's actually a D.O. IRL, so this is 100% hobby for him.
Re:How Google Hires by hoboon on Tuesday April 07, 2015 @07:52PM (#9337952)
> I think the best way to get over that is to make something on your own that is so great that everyone wants to hire you without even doing an interview!

This is a good idea.

I have no idea what people would want or what I could make that people would want or what open problems are there that I could make. Every idea I have someone else already has a better solution.

I keep trying and put it in github but it's mostly for me; no one looks at it.

Re:How Google Hires by RogerL on Tuesday April 07, 2015 @10:15PM (#9338450)
Okay, here is an idea.

The Julia people are complaining that their language is great but it is not being picked up because their standard libraries are missing a ton of functionality.

Start knocking out some standard library functionality.

Sure, you may not be interested in Julia. But there are plenty of other projects out there that are understaffed. IPython Notebook needs developers. Octave needs somebody to write a good front end. It goes on and on.

The advantage is that you don't have to be super creative (compare Julia's libraries to Python, and start implementing something that isn't yet in Julia), nor predict the future (meaning, you can make a brand new X, but if the world goes to Y, you will never get picked up). What you write will be used by (at least) thousands, and you will have to write production quality code to get your pull requests accepted. I'd be impressed with anyone that did that, even if I had no interest/need in the project that they contributed to.

Re:How Google Hires by danjayh on Wednesday April 08, 2015 @12:28AM (#9338788)
I would like to proffer a different strategy: Pick something that you are passionate about, good at, and comfortable with. Then push your personal boundaries and learn something new while making some app/widget/whatever, even if it's been solved/done before. Why did you do it? You wanted to stay up to date, learn something, and it was fun (and all of this will be true).

Whatever you made will probably be something pretty sweet because you chose something close to your heart, not something that was simply 'unsolved' or 'needed doing'.

Employers will love that you chose to challenge yourself to build new skills as a hobby, and because you chose something that you are passionate about, you'll probably also have a good amount of enthusiasm while you talk about it, which is equally important.

Re:How Google Hires by cgearhart on Wednesday April 08, 2015 @10:33AM (#9341004)
That sounds like a great idea. Do you have any specific links or resources regarding limitations of the Julia standard library?
Re:How Google Hires by proksoup on Tuesday April 07, 2015 @07:59PM (#9337985)
Where is your github? People bothering to respond to your comments on HN might be the only ones you will be able to get to glance at your code ... but I bet they will. I would have if I had been able to find your github.
Re:How Google Hires by jedanbik on Tuesday April 07, 2015 @08:02PM (#9337991)
Are you showing it to anyone? Put it in your profile, for starters.
Re:How Google Hires by jmccree on Wednesday April 08, 2015 @11:16AM (#9341271)
Pick an open source project that interests you, preferably one you personally use, and become active in their community. The issue/bug tracker of almost any open source project has a list of feature requests and bugs to work on. You get to add open source contributor to your resume, and become a "subject matter expert" by virtue of that.
Re:How Google Hires by proksoup on Tuesday April 07, 2015 @07:58PM (#9337981)
You've got about billion comments like this in your history.

The previous advice has been good.

If you opened up more about your specific troubles I think more on HN might help diagnose.

Maybe put some stuff on github, who cares how awful it is or if no one will use it .... The following statement is not globally true but I think it is probably true in your case --- you won't be able to succeed until you fail some more.

If there was a "hoboon" github we could like at when responding to your comments, the specific advice might be more specific.

Re:How Google Hires by hoboon on Tuesday April 07, 2015 @08:04PM (#9337999)
You're right. I talk about this often. I'm going a little crazy, and maybe angry.
Re:How Google Hires by proksoup on Tuesday April 07, 2015 @08:11PM (#9338021)
Yeah yeah, it's all good. We cool.

Show us your code.

Re:How Google Hires by chernevik on Tuesday April 07, 2015 @10:05PM (#9338422)
Frustrated and wanting to improve your skills -- fine.

Hate yourself -- hmm, not good.

Everyone struggles and it's very hard. Looking in the mirror and figuring out what you can do better, not being satisfied with what you've achieved, I'm all for that. It's how we grow. But you cannot allow your assessment of yourself on a single dimension to become your entire assessment of your entire worth. I'm certainly not saying we're all special flowers, and our professional is hugely important. But it isn't all we are, and those skills aren't static anyway.

Also: If you want feedback, you can do things to improve your chances of getting it. Be incredibly positive in your responses -- "thanks for the opportunity!" People are obviously reluctant to talk with someone they've rejected, so signalling that you won't make this hard for them can help. If you have guesses at the problem, or suspicions you'd like to rule out, suggest them. Statements like "I think I need to build up Z" or "I worry that I come across as Y" might give you useful yes / no answers, if they are set up properly.

Think also about informational interviews as a means for seeking out fit. If you're just having coffee with someone, about what they look for and how people succeed, they are much more likely to be brutally honest than they are after rejecting you. If I'm having coffee, I can politely tell you why your resume or body of work look off, because I think that's helping you.

Re:How Google Hires by kelukelugames on Tuesday April 07, 2015 @08:43PM (#9338122)
May I humbly recommend this resource?

https://www.ocf.berkeley.edu/~kelu/interviews/

Re:How Google Hires by Decade on Wednesday April 08, 2015 @04:38AM (#9339383)
Don’t hate yourself. I refuse to hate myself. I know my many skills and powerful talents. Hate the system.
Re:How Google Hires by 72deluxe on Wednesday April 08, 2015 @04:50AM (#9339433)
Where do you live?
Re:How Google Hires by kelukelugames on Tuesday April 07, 2015 @03:07PM (#9336314)
Regarding snap judgements, I've had multiple interviewers write me off within the first five minutes. It's just silly and a waste of everyone's time.

Example: Interviewer asks me a question. I start brainstorming outloud because they coached me to talk through my ideas. Of course some of my ideas are not going to work. The interviewer will shoot the first wrong idea down immediately and peg me as an idiot for the rest of the interview even if my second idea is the perfect solution.

Re:How Google Hires by IndianAstronaut on Tuesday April 07, 2015 @05:45PM (#9337380)
This is the exact opposite of an interview I used to run in my old biology lab. We used to sit down a new candidate in front of a microscope and ask them to do some cell transfers and counting. They had to finish the task.

We right away knew if they were persistent, skilled, problem solvers, and if they were meticulous.

Re:How Google Hires by serve_yay on Tuesday April 07, 2015 @07:28PM (#9337859)
If they pulled it off maybe you knew that. If they didn't -- is it because they suck? Or because they're incredibly nervous and just got put in an environment which which they were not acclimated, and told to do work?

I may not be a scientician, but I'm pretty sure there's something in the Big Book of Science about controlling for variables...

Re:How Google Hires by IndianAstronaut on Tuesday April 07, 2015 @09:37PM (#9338315)
They may have been good, but we have no way of knowing that if they are nervous. I doubt too many employers have this figured out either. If the candidate is nervous and doesn't perform, there isn't too much that can be done about it other than maybe looking at past history.

We were also rigorously testing to see if they could get acclimated to the environment. They would have to do this routinely since there were so many new lab techniques to be learned.

Re:How Google Hires by cozuya on Tuesday April 07, 2015 @03:13PM (#9336373)
I think its the opposite. I would much rather had only my and the candidate's 5 minutes wasted, versus a whole hour, when its very apparent in those first 5 minutes they don't have the technical skills we need for the job.
Re:How Google Hires by tptacek on Tuesday April 07, 2015 @03:15PM (#9336390)
Presumably you consider how drastically your whole process has failed if you bring someone in for an on-site interview who demonstrates within 5 minutes that they lack all the technical skills required for the job.
Re:How Google Hires by cozuya on Tuesday April 07, 2015 @03:36PM (#9336575)
Well mistakes happen - a phone screen can only get you so far, especially in a seller's market where your talent pool is pretty shallow. Though I think a shared google docs interview is probably the best way to go for the initial screen, I just haven't given or taken one like that yet so it must still be relatively rare.
Re:How Google Hires by tptacek on Tuesday April 07, 2015 @03:38PM (#9336588)
We came to the conclusion that phone screens are pretty close to worthless. We'd still have been very upset if we ended up in a face-to-face interview with someone who had no business interviewing for the job. You need to do more than just phone screen (actually: you should eliminate phone screens).

(I don't know who's downvoting you but I upvoted.)

Re:How Google Hires by JohnBooty on Tuesday April 07, 2015 @04:02PM (#9336763)
I'm really fascinated by your replies on this thread; seems you build a really solid process there.

I'm curious about the phone screens - how do you screen out the candidates that are grossly unqualified so that they don't progress to the interview stage?

In other words, how do you screen out candidates like those Joel describes here: http://www.joelonsoftware.com/articles/ThePhoneScreen.html

Re:How Google Hires by tptacek on Tuesday April 07, 2015 @04:07PM (#9336792)
By my last year at Matasano we were literally discarding the results of phone screens. We did them largely as a pro-forma exercise, and because they made team members more comfortable with the process as a whole.

We virtually never selected out candidates based on phone screens. We had a work sample process that kicked in after phone screens, and that cost us almost nothing to run. So we never had an incentive to prevent a candidate from going through that process.

Unlike phone screens, the work sample results were strongly predictive. You could bomb phone screens, ace work samples, and end up coming in for in-person interview.

(We did "nudge" candidates we felt wouldn't do well on the work sample tests, solely out of concern for not wasting their time, but anyone who wanted to proceed was able to).

Phone screens are a waste of time.

Spolsky did a really good job of documented the best practices within the framework of unstructured interviews as they were practiced 10 years ago. The problem isn't Spolsky's tactical suggestions; it's that the strategy they're a part of is being discredited.

Re:How Google Hires by timv on Wednesday April 08, 2015 @03:24AM (#9339192)
I'm curious how often you had people voluntarily drop out of the process without completing the work sample, and whether that frequency would change if you didn't have a phone screen. (I take it from your questions that you never actually stopped doing phone screens).

If my first contact with a company is them asking me to put in hours of my time, when it costs them "almost nothing to run" (your words) then I'd be inclined to pass.

The company I work for has a work sample exercise, and we intentionally place it after a phone screen and first interview, because we feel it is (and appears) more fair to the candidates.

From our point of view, we'd love to put it up front, as it is the best source of information we get, but if it caused us to lose good candidates before we even started (and we believe it would) that would be a show-stopper.

Re:How Google Hires by jerematasno on Monday April 20, 2015 @03:06PM (#9410165)
Most of our candidates drop out before the work sample. On the other hand, almost none of our candidates are qualified to work for us when they initially apply.

We make it really clear that there will be work samples before people even apply. I think the real question would be "how many qualified applicants don't bother applying because of the work sample," which is a question we can't answer. Given the paucity of unemployed qualified infosec folks, we're comfortable with the tradeoff.

Re:How Google Hires by Decade on Wednesday April 08, 2015 @04:58AM (#9339460)
I’m curious exactly how many times a candidate gets to apply to Matasano.

Thomas describes the Matasano process as costing “almost nothing,” but that includes running a couple exploit training websites and sending an 852-page book to applicants. Which I’m not inclined to go through immediately, because I need money right now.

Do I have to do the microcorruption.com and the cryptopals.com and The Web Application Hackers Handbook before even trying the technical screens and challenges? Or should I try seeing if my existing web application security best practices and rusty MIPS assembly experience are enough to get to where I have enough breathing room to do these exercises?

Re:How Google Hires by jjarmoc on Thursday April 09, 2015 @05:01PM (#9351004)
> I’m curious exactly how many times a candidate gets to apply to Matasano.

Undefined. Personally, I applied twice, ended up coming to work here the second time through.

> Do I have to do the microcorruption.com and the cryptopals.com and The Web Application Hackers Handbook before even trying the technical screens and challenges?

Nope. But they might help, and you may want to anyhow; they're fun :)

Re:How Google Hires by tptacek on Wednesday April 08, 2015 @08:58AM (#9340402)
No! You absolutely do not need to do Microcorruption or Cryptopals.
Re:How Google Hires by vonmoltke on Wednesday April 08, 2015 @10:16AM (#9340888)
On the other hand, if you made something you could give to anyone who applied and was little to no work for you how many good candidates would you pick up that you wouldn't have given a phone screen to?
Re:How Google Hires by timv on Wednesday April 08, 2015 @09:40PM (#9345346)
I think that varies quite a lot based on company profile and role-type.

We phone screen anyone who has a resume that's vaguely relevant. If you're applying for a software engineering role and have never done any development, then we'll reject straight away, but that almost never happens.

We simply don't have a high enough profile to receive a flood of applications that we need to filter before we phone screen.

And for a software engineering role, evaluating the work test takes as much as the phone screen (sometimes more). We're not simply checking whether you can produce the right output for FizzBuzz (checking that could be automated), we're looking at your choice of algorithm, design trade-offs, unit testing approach, etc - things that require a human to be involved.

Different companies and different roles will have different time investments.

A CTF style task can be evaluated more simply (at least on first pass) - did you capture the flag.

And if you are looking for a few amazing candidates form amongst a mass of poor ones, then you can afford to optimise your process so that you test every candidate at as low a cost as possible, even if that causes some candidates to self-select out.

Re:How Google Hires by collyw on Friday April 10, 2015 @06:31AM (#9353446)
I have heard much the same about dating, so I don't think it is necessarily about a process failing, as the way we judge people based on first impressions.
Re:How Google Hires by nostrademons on Tuesday April 07, 2015 @03:35PM (#9336561)
Curious how you could judge that a candidate doesn't have the technical skills you need for the job in 5 minutes? Can all the tasks your employees do on the job be completed in 5 minutes?
Re:How Google Hires by JohnBooty on Tuesday April 07, 2015 @04:04PM (#9336778)
Sometimes you get applicants that literally don't know what a compiler is, don't know what a variable is. That sort of thing. That kind of person can probably be ruled out in the first 5 minutes for sure.

Of course I think everybody would agree that if such a person progressed to the interview stage, there was a failure somewhere.

Re:How Google Hires by jotux on Tuesday April 07, 2015 @03:31PM (#9336524)
Thanks for your hiring post article -- it's one of the best things I've read about what interviewing should be and I've shared it with countless friends and colleagues in the last month.
Re:How Google Hires by tptacek on Tuesday April 07, 2015 @03:35PM (#9336566)
Thank you!
Re:How Google Hires by beambot on Tuesday April 07, 2015 @07:17PM (#9337815)
> Our structured interview process differed from Google in that we (a) completely standardized interviews

Google interview questions are leaked so often that standardization is virtually impossible...

Re:How Google Hires by derefr on Tuesday April 07, 2015 @08:13PM (#9338027)
I don't see why it's so hard to just standardize on a formula for generating questions, rather than a static question.

There's a simple method of doing this that doesn't require hard AGI to generate the questions, either: just standardize on a list of the microskills the job requires. Then, get the interviewer to create a small question to test each microskill. Keep previous interviewers' questions together with the microskill in the list as examples for the next interviewer.

Re:How Google Hires by RogerL on Tuesday April 07, 2015 @10:08PM (#9338435)
Why does that matter?

Your question isn't very good if you are selecting for people that know your question vs are smart/skilled.

Re:How Google Hires by samstave on Tuesday April 07, 2015 @07:53PM (#9337954)
Even if I were not fit for a job with you, or a position in your company, I would LOVE to interview just to experience what this is like.
Re:How Google Hires by mbesto on Tuesday April 07, 2015 @09:56PM (#9338391)
> (a) completely standardized interviews and (b) designed the questions in those interviews to minimize the need for free-form follow-up questions

Doesn't this just make the system easier to game? It's like SAT prep. SAT scores correlate with the amount of time/money preparation for it, not your actual knowledge.

Re:How Google Hires by kevingadd on Wednesday April 08, 2015 @01:19AM (#9338907)
The process of answering matters far more than the answer. It's not a graded multiple-choice test.
Re:How Google Hires by simula67 on Wednesday April 08, 2015 @04:02AM (#9339283)
Out of curiosity, how do you know if your interview process is better ? Is there some scientific method ? A lot of what you write makes sense, but how can I convince myself that I am not falling prey to Confirmation Bias ?
Re:How Google Hires by benihana on Tuesday April 07, 2015 @03:43PM (#9336631)
I really enjoyed this article, thank you for writing it. I'm going to marinate on the things you talked about and see if I can try to influence some change where I work.
Re:How Google Hires by tptacek on Tuesday April 07, 2015 @03:46PM (#9336649)
Let me know if there's anything I can do to evangelize. We got this to work, and it was pretty amazing to watch.