News for hackers. Stuff that matters.
Re:Good at programming competitions does not equal good on t... by williamsmj on Friday July 07, 2017 @03:01PM (#14721108)
It's not that simple.

Many good engineers, especially those who aren't young men, don't have existing work samples they are able to share. Perhaps, by choice or otherwise, they have a personal life that does not allow them to write code for free. And their employers won't allow them to share work code.

And many good engineers are not willing to spend an unpaid weekend writing code auditioning for a job they aren't yet sure they want.

And, as you know if you've ever worked with a disruptive colleague, or had customers, engineering in a team for a product is much more than just writing code.

Re:Good at programming competitions does not equal good on t... by tptacek on Friday July 07, 2017 @03:08PM (#14721176)
That's not what a work sample is.

A work sample is a (usually, hopefully) standardized piece of work you request from all candidates that mirrors the actual work they'd be doing on the job.

There's no reason a work sample needs to be so onerous that it costs you a weekend. What people forget when this comes up for discussion is that interviews are work. In fact, they are themselves onerous work, since they require you to be on-site and intensely engaged in ways you don't have to be to do the actual work of a job.

A work sample, on the other hand, can be done from your home, with a beer next to your laptop if that's how you roll.

The notion of audition work has definitely been abused in our industry. People get work samples that aren't standardized. They get work samples that are later ignored. Work samples don't have objective scoring rubrics. Some annoying companies assign new features for their products as "work samples".

Done carefully, though, with objective and predictable grading and calibrated to offset in-person interviewing time, they're superior to any process we have.

Re:Good at programming competitions does not equal good on t... by ryandrake on Friday July 07, 2017 @03:44PM (#14721460)
As an applicant, I'd be all for going the work sample route, but only if it's paid for and if it were used in lieu of the typical face-to-face hazing session rather than in addition. I notice more and more companies are asking for work samples (or "homework assignments" or whatever they want to call them), but they just pile it onto their already onerous process.

The major pain point in interviewing (at least for me) is the massive time sink for a very small chance of success. I'm spending a few hours on the application, then a few more doing phone screens, then a weekend doing a work sample, then I blow a VACATION day (which has monetary value and of which I have only 10 per year), and at the end of it all who knows what my odds even are? Multiply that by, lets say 10 companies per year, and: I've invested 31 days of my life, have no vacation left, and I still may end up with nothing.

Re:Good at programming competitions does not equal good on t... by tptacek on Friday July 07, 2017 @03:51PM (#14721506)
I think paying for interviews is problematic for a bunch of reasons.

On the other hand, I agree wholeheartedly with people who reject "homework assignments" layered on top of a standard interview process.

The onus is on people who want to take advantage of work samples to:

* Ensure that the time they're asking of candidates is offset by lowered time demands elsewhere

* Ensure that they're using the work samples objectively, so that people aren't asked to do coding work as part of a crap-shoot application process.

The process we used at Matasano and NCC was less demanding than typical job interview processes. The challenges were simple and self-contained, and when they were completed we could tell you with a decent degree of confidence whether you were likely to be offered a job at your (shortened) on-site interview.

Re:Good at programming competitions does not equal good on t... by bdcravens on Friday July 07, 2017 @04:36PM (#14721850)
> I still may end up with nothing

What do you end up with if you filter out companies that ask for work samples?

Re:Good at programming competitions does not equal good on t... by ryandrake on Friday July 07, 2017 @04:42PM (#14721896)
1 additional free weekend for every one I don't do.
Re:Good at programming competitions does not equal good on t... by bdcravens on Friday July 07, 2017 @04:46PM (#14721932)
I'd definitely say that if a company is handing out work product assignments that take an entire weekend, it's somewhere you don't want to work.
Re:Good at programming competitions does not equal good on t... by williamsmj on Friday July 07, 2017 @03:12PM (#14721207)
As I said, many good engineers are not willing to spend unpaid time auditioning for a job they aren't yet sure they want, doing work they can't reuse, by taking a formal standardized test.

It may in fact be a better hiring practice in terms of its ability to predict job performance. But you will _lose good candidates_.

And in particular, you will lose experienced candidates. You are embedding biases toward younger (and therefore less experienced) male coders in your hiring practices if you require what is, from their point of view, free work. (You can see the biases in your "with a beer next to your laptop" remark, for example.)

Re:Good at programming competitions does not equal good on t... by bit_logic on Friday July 07, 2017 @03:41PM (#14721434)
I'm an experienced engineer (10+ years) and I would favor this process way more than the current algorithm/DS interview. It's the algorithm/DS interview that really favors young graduates. They already have an advantage because college knowledge is still fresh in their minds. And they often have the free time to grind leetcode for months. Experienced engineers often have families or other priorities and free time is hard to find.
Re:Good at programming competitions does not equal good on t... by tptacek on Friday July 07, 2017 @03:15PM (#14721226)
You updated your comment after I replied to it, and so I updated mine.

Source: I ran a work-sample recruiting process for the largest software security company in the US (after it acquired my startup, which was one of the top 5 in the US), and our process most definitely was not biased to younger workers. Work samples drastically improved diversity.

Edit

You've edited your own comment again, which makes you very hard to respond to. If you want to rebut anything I've written so far, can I ask you to copy it into a reply to this comment?

Re:Good at programming competitions does not equal good on t... by williamsmj on Friday July 07, 2017 @04:09PM (#14721647)
Sorry, new commenter, unfamiliar with the etiquette of the edit button. I'm done.
Re:Good at programming competitions does not equal good on t... by chucksmash on Friday July 07, 2017 @03:22PM (#14721274)
I agree with you somewhat, but couldn't you just as easily replace "unpaid weekend writing code" with:

- unpaid weekend updating resume

- unpaid weekend practicing algos

- unpaid weekend browsing job sites

just as easily? I don't think asking job seekers to be interested enough to give a work sample is an undue burden. If that filters out people playing the numbers by applying to every opening...is that such a bad thing?

Re:Good at programming competitions does not equal good on t... by morgante on Friday July 07, 2017 @04:48PM (#14721944)
If I spend a weekend updating my resume, practicing algos, or browsing job listings, then that time is useful for all jobs I'm applying to and interested in.

If I spend my weekend on a work sample assignment, it's only applicable for that one company which might very easily not give me a compelling offer or be somewhere I'd want to work.

At this point, I'll only do work assignments if I'm already very familiar with the company and have reason to suspect they'll give me a compelling offer.

Re:Good at programming competitions does not equal good on t... by ng12 on Friday July 07, 2017 @03:58PM (#14721569)
It's a problem of scale. Spend half an afternoon skimming through CtCI (shouldn't need much more as a seasoned developer) and you're good for N number of job interviews. I'll take that rather than spend 4*N hours on take-home problems in addition to time spent on-site.
Re:Good at programming competitions does not equal good on t... by aNoob7000 on Friday July 07, 2017 @03:34PM (#14721380)
It depends on how far along the interview process you've gotten to with a potential employer and if this is a step away from being hired.

If the potential employer is asking for code samples up front, I think that's a bad sign. Let us talk about the work environment, expectations of the position, and then we can focus on my tech skills.

Re:Good at programming competitions does not equal good on t... by poletopole on Friday July 07, 2017 @03:43PM (#14721449)
Usually the work sample is something simple that doesn't take longer than a day or two. Many interviews that don't do work samples can go on longer than that and accomplish nothing. I was hired after doing a work sample that took a day. I was still paid for my work, irregardless of being hired. The work sample wasn't "standardized", just a typical task one of the other developers would have done otherwise. So I would encourage any employers reading to try this process out--it's better for both parties.
Re:Good at programming competitions does not equal good on t... by desiderantes on Friday July 07, 2017 @03:47PM (#14721481)
>You are embedding biases toward younger (and therefore less experienced) male coders

Younger maybe, but where does the male part come from?

Re:Good at programming competitions does not equal good on t... by williamsmj on Friday July 07, 2017 @04:07PM (#14721632)
Less likely to have personal obligations to others, e.g. family. Less likely to feel social pressure to do time-consuming emotional labor outside of work.
Re:Good at programming competitions does not equal good on t... by tptacek on Friday July 07, 2017 @04:18PM (#14721710)
How is the standard interview process, which often requires candidates to submit to multiple rounds of in-person interviews, sometimes even requiring travel, better than a process that people can do mostly from their homes?
Re:Good at programming competitions does not equal good on t... by methodover on Friday July 07, 2017 @07:15PM (#14722971)
Honest question:

I'm the tech lead for a startup with just a handful of programmers. I'm interested in hiring someone who can help us come up with well-architected, novel solutions to the problems that we're already solving. The issue is, we're solving them in a way that turns out to be pretty difficult to maintain and change. I'd like a totally new, fresh approach that takes the lessons we've learned and creates something that's much easier to write fixes for, much easier to extend with new features.

I need someone who doesn't just build on top of existing, well-defined codebases/APIs. I need someone who has the ability to engineer a complete, well-rounded, extensible codebase/API himself.

It would seem to be remarkably difficult to test for this ability. Even work samples that can be done in the short time period you describe wouldn't be able to test for this ability.

Re:Good at programming competitions does not equal good on t... by tptacek on Friday July 07, 2017 @07:21PM (#14723006)
Short work sample tests don't need to be simple work sample tests. One of Matasano's work samples required candidates to reverse-engineer a binary protocol, build their own client for it, and then use that client to find vulnerabilities. If you had no aptitude or experience with that problem set, finishing it would take a very long time.†

Ultimately the problem you're really describing is the challenge of hiring people you don't really know how to hire. If you know very well how the role you're hiring for is supposed to function, you should be able to generate challenges within whatever constraints your candidates have. If you don't know very well what the capabilities of that role need to be, no interview process is going to work reliably for you.

† (we didn't care, by the way: if you did well with that work sample, it mattered very little to us whether you were prepared when you started it).

Re:Good at programming competitions does not equal good on t... by hex13 on Saturday July 08, 2017 @12:26AM (#14724078)
"build on top of existing, well-defined codebases/APIs." is not necessarily easier than "engineer a complete, well-rounded, extensible codebase/API himself."

I think these are two different skills. Some programmers are better in building on top of existing codebases, some programmers are better in build things from scratch.

maintaining vs starting.

Re:Good at programming competitions does not equal good on t... by potatolicious on Friday July 07, 2017 @03:14PM (#14721222)
I don't mean to propose that people spend their off hours writing work samples - that's unreasonable for all the reasons you've given and more.

What I mean to propose is that we dramatically alter the current on-site interview process used by most companies.

Instead of 5-7 interviews, consisting of 5-7 independent but ultimately equally contrived algorithms questions on a whiteboard, use the several hours you have the candidate to produce real, working code that reflects the work that your company does.

Which is to say, this requires no more time commitment than the existing interview processes at typical tech companies.

If the work your company does is deeply algorithmic, this will be reflected in the work sample produced via this process. If the work your company does is more heavily UX-oriented, this will be reflected in the work sample produced, also.

Instead of "over the course of a full-day on-site round of interviews, candidate produced several short snippets of hand-scrawled code solving various CS textbook problems, which we will now use to infer general engineering ability"...

... you can say "over the course of a full-day on-site round of interviews, candidate designed and produced a module of code that does [X small thing that company does], which can be assessed with the same rigor and methodology we already apply to our existing work and employees"

Re:Good at programming competitions does not equal good on t... by sidlls on Friday July 07, 2017 @03:54PM (#14721532)
While I agree that a focus on "work samples" is inappropriate, I must emphatically agree with the parent's broader point.

The way companies in this industry conduct interviews and measure performance is like an aerospace engineering company testing candidates for satellite engineering on their understanding of the Standard Model (physics) or group theory or some such. There's too narrow a focus on what is required of an engineer in the software industry.

Re:Good at programming competitions does not equal good on t... by softawre on Monday July 10, 2017 @01:44PM (#14738097)
You are allowed to pay somebody for a work sample.
Re:Good at programming competitions does not equal good on t... by angersock on Friday July 07, 2017 @03:08PM (#14721174)
> don't have existing work samples they are able to share.

This, at least, should be fixable.

Don't work at places that won't let you talk about your work and demonstrate your competence.

Re:Good at programming competitions does not equal good on t... by pm90 on Friday July 07, 2017 @03:34PM (#14721379)
I don't think that's always possible. What if you worked for Healthcare or Security or Finance, where there are strict laws for keeping code/design private?
Re:Good at programming competitions does not equal good on t... by angersock on Friday July 07, 2017 @03:55PM (#14721535)
What are examples of such laws? There are laws allowing prosecution if you break agreements, sure, but the agreements themselves are what can be fixed by encouraging the market that they're incorrect.
Re:Good at programming competitions does not equal good on t... by DanBC on Friday July 07, 2017 @04:10PM (#14721655)
Re:Good at programming competitions does not equal good on t... by angersock on Friday July 07, 2017 @05:55PM (#14722477)
That seems targeted at .gov employees, which is a little different.