> Well, empirically we're getting quite a lot of people attempting it!

> Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other.

So instead of hundreds or thousands of unstructured resumes in different formats, you have hundreds or thousands of completed programming tests. Even assuming that you automatically filter out submissions that fail testing, given that there are a lot of ways to solve a problem, you'd still be stuck evaluating potentially hundreds of correct submissions.

More importantly, a test doesn't seem to be aligned with your top priority:

> ...but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success.

Are you assuming that all of the folks most accomplished in the real world are going to complete your programming test successfully?

Since you ultimately seem to be interested in folks who can build dashboards, I would think you'd be far better off asking candidates to build a dashboard prototype using sample data you provide. There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application.

> You'd still be stuck evaluating potentially hundreds of correct submissions.

I'm guessing the MVP of hackerrank was the web code editor + lint/findbugs/whatever for filtering. A lot of people have spent a lot of time writing open source projects that convert code to metrics, so making a reasonably prioritized list is probably fairly easy.

> There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application.

Agree!

I think this is a great critique of HackerRank as a business idea -- while they are new, it's just a hurdle that weeds the busier coders out of your hiring pool... if they become established there will be books on cramming for it, and will no longer separate the wheat from the chaff.

(ie https://web.archive.org/web/20130114163457/https://raganwald... meets http://en.wikipedia.org/wiki/Goodhart's_law)

To be honest I think it's the other way round - if you can do programming tests all day long you will succeed at most coding jobs. However there are a lot of developers who can't tackle an interview that with a nights sleep and no pressure can easily code a intuitive and effective architecture.

Sorry - programming tests raise far more false negatives than false positives (and this from a person who considers himself a false negative in most tests)

I'm not sure what kind of coding work you do, but I assure you the vast majority of web development work has little resemblance to coding questions. You are almost never designing an algorithm: that was already done for you by whoever designed the library you are using.

Majority of the tasks IME:

- code reading & refactoring

- figuring out why two libraries aren't playing nice

- bug triangulation

- optimization (not the algorithmic type, but rather the fix-the-auto-generated-SQL-query / frontend asset packing kind)

- repetitive/simple/obvious bits of jQuery

Maybe there's some kind of online test that could exercise those skills, but if you ask me the main performance changers for modern web development are, they're: really knowing the stack (problem anticipation, what to use, how to use the various tools to quickly get search keywords), google fu, and most importantly having done it a couple hundred times (muscle memory).

That's terribly negative. If you are doing agency work maybe (same website, same functionality, just different skin) but that vast vast open plains of http-based development are around and going to be around joining parts of the business, public world together in ways they do not know or imagine.

Yes, binary trees are pretty much coded for you, and you would be a fool to roll your own hashing algorithm for production, but thats still not a reason to keep your head down.

The next decade is going to be fun. Trust me

I don't understand this response. To be clear, I think web development is fun. I very much enjoy it and don't resonate with "keeping my head down".

I guess to reiterate my point: once you've done web development for a couple years, coding most websites is less about solving new problems and more about applying known solutions.

To reiterate how this point relates to the question of coding interviews: in as much as they don't test for lots of known solutions, and the ability to apply known solutions quickly, they aren't filtering for productive coders.

How does your response relate to the points at hand?

but web development is not about "making new websites"

Think micro-services supplying REST APIs over HTTP - that's going to be same-y after a few years yes, but behind that is a world of business apps, integration challenges, complex algorithms and calculations - sorry I just see "web development" as "anything that has of will have an http access point" which I mentally count as "pretty much everything from here on in"

Programming tests do generally test for ability to apply known solutions quickly - it's about all they do do. You just have to know the known solutions (oh look calculate primes).

As for finding productive coders - well the amount of false negatives is large, and the false positive rate is probably higher (plenty of people who can waltz through these tests i find fairly poisonous to morale) so I am not a big fan - but if you need to make a first cut this is a good start

Another key:

- Communicating clearly and calmly with other people

You're right, we need to add social skills and domain knowledge to the list.

> Well, empirically we're getting quite a lot of people attempting it!

I think you missed my point. It's that the people submitting to a multi-hour exam just to submit a resume are more desperate / value their time less than those that can stand on their experience.

The fact that there a "quite a lot" of submissions only worsens the probability that the multi-hour effort will be wasted.

The questions were pretty good, because they were close to the real world and you couldn't google them so easily. The programming tasks were rather unbalanced. The first two were super easy, while you can easily get stuck on the last one, even as a good programmer. I knew that I wouldn't be able to answer it today, so I skipped it after a short while. If I'd plan such a test, I'd try to get an example of the structure and comprehensibility of my future employees work.

Do you mind if people do the test with no interest in the job?

Sure, go ahead! We just ask that you not share the questions with others after completing it.

>> allows someone ... from outside the US, to compete on a level playing field.

Just curious, why do you say that the normal process doesn't allow people outside the US to compete on a level playing field? Higher barrier to on-site interviews?