1. Reading and evaluating code in an unfamiliar context is really £@#&ing hard, especially on a complex project. Understanding someone's commits requires knowledge of the project as a whole, the problem space, the environment, etc. That's not possible to do for a large number of candidates. It might not give you a very good insight in to an applicant.

2. Most people's public code is of roughly equal quality (eg 'not great'). You can't really differentiate between people well based on it. What people do in their spare time for fun is rarely commented/tested/documented/etc, so it's a bad way to evaluate their skill level. If you find someone with good public code that's a great sign, but a hiring manager who's looked at 100 Github profiles and found zero good ones might reasonably have given up looking.

Also it could be seen as a discriminatory factor; if you think public code is important you're going to overlook people with little time to write any, eg those with families, care responsibilities, interests outside of coding. Building a great team is about hiring great people more that just great developers.

3. It's hard to trust someone's public code. It's trivial to clone a few open projects and take credit for the work, and a hiring manager doesn't have the time to dig in to everyone's projects to see if they've ripped the work off from somewhere.

Also, you may have dealt with 50 hiring managers who have never looked at your public code in your career so far. That isn't very many really.

I certainly wish there was a better way to evaluate developers skill, and as someone who has plenty of public work on Github I think it'd be great if that was a good starting point, but I don't think it's quite good enough yet. I can certainly see why hiring managers don't look there.

I'd argue pointless whiteboard trivia doesn't give you good insight into an applicant, either. And yet, 90% of companies do it.

> Most people's public code is of roughly equal quality (eg 'not great').

I disagree with this. Unless you're looking at a toy or unfinished project, most code that's released (on npm, maven, or something similar) is actually fairly high quality. Certainly higher quality than what I've had to work with in production systems that were making millions of dollars a month.

> Building a great team is about hiring great people more that just great developers.

I completely agree. So why do we dehumanize potential hires with standard practices that involve (for example) memorizing algorithms you'll never use? I mean, HackerRank is a testament to how awful our culture is. Is there a DoctorRank or SecurityGuardRank or WaiterRank?

> It's hard to trust someone's public code.

I think this is complete BS. There's people much smarter than me that are getting screwed by these terrible practices. Max Howell, Homebrew's author, was famously rejected by Google for not passing some stupid whiteboard exam, when just about everyone at Google actually uses the software he authored.

The whiteboard interview was to write out algorithm for a merge sort. I thought it was the silliest thing in the world. I had already accepted a position to basically create a development department, and he wants me to write a sort algorithm? Once he described how it worked, I wrote it without much difficulty but that was a turn off and I got an offer that was permanent (as opposed to contract to perm for the offer I accepted), making more money, with a signing bonus.

It was such a turn off, I declined the offer for a chance to build a department that would not expose developers to this foolishness and thought that the interview process tells you a lot about the company.

A year later. I'm glad I made the choice I did.

On the other hand, I've done quite a few pair programming type of interviews where there was a skeleton of a program with a lot of failing unit tests you had to make pass. I enjoy those.

Is there a DoctorRank or SecurityGuardRank or WaiterRank?*

Doctors have to go to school 8+ years and do a 2-4 year residency before they can practice without supervision. Lawyers have to go to school 7 years and pass an exam. Software Engineers only have to watch a couple of YouTube videos and they can call themselves "programmers". Doctors can't say they are qualified because they have been doing brain surgery since 6th grade in their bedroom.

At this stage in my career when I still get this type of bullshit from someone I just find it ridiculous.

I'm just left thinking, "Did you read my cover letter? Where I told you how I worked on a 3 million CLOC air flight cockpit software that runs in actual planes; so if there are bugs real people would die? How about the part where I discussed being the software lead on financial derivatives trading software; where bugs could lead to millions or even billions of dollars be lost? Did you see that software library I maintain in my free time that has over 1 million installs? And you still aren't sure I'm qualified? Well no sense wasting any more of your time; I'm clearly not qualified then."

Ironically what I would consider the two best interviews of my career went like this respectively.

1) I was called in for a technical interview. The VP of the IT department sat down with me and asked me to tell him about my work experience. I gave him an overview and then he started delving in deeper and deeper on the subjects that I said I had experience in to see what my level of knowledge was. This went on for about 1 1/2 hours at which point he stopped me and said, "Ok, how much money do you need?" I gave him a number to which he replied, "We could do that. Hold on a minute." The VP then went and brought in his top developer (turns out he's also the hiring director) and then we basically repeated the whole thing, but dove into even more technical details. That lasted somewhere between 2 and 3 hours. At the end of it this he says, "I can have an offer letter ready for you tomorrow. When can you start."

2) Preface - I was referred by someone already working with this company.

I get a phone call from the company CEO (small start-up). He asks me a few questions about my background, some soft skill type of questions, nothing real technical. After 20 or 30 minutes of conversation that is basically just a "can we get along with each other" check the CEO says, "Ok, I'll need you to do a technical interview with our team lead." A couple days later I get a call from the team lead.

The conversation goes basically like this.

Him, "Ok, so here's the deal. Steve says your really fucking smart and Steve is really fucking smart so it's probably true. Elijah (CEO) thinks we need to do technical interviews, but really they're all just bullshit anyway. If it turns out you can't do the job we'll fire you in less than three months; if you actually know what you're doing we won't. So do you want the job or not?"

Me, "Yeah"

It was a contract to perm position. I asked my now manager, why was this a contract to perm position? Was it because they wanted to see if the person was a good fit or were they under a budget constraint that might stop me from coming on permanently? He said the former.

I had to create the department from scratch with basically nothing but "database developers" who were just learning C# and no infrastructure. I laid out a 6 month plan. Six months in, i was able to negotiate a higher pay than I could have originally since it was my first official lead role and make extra money by being able to bill hourly. It was a major win.

When I was single without the skill set, network, or health benefits from my spouse, I would have never left a full time position for a contract to perm.

Even if there was, no one would trust it. Case in point: ABET will accredit CS programs like they do other engineering programs; not many schools bother because companies don't give a shit about it.

> You can have a CS degree without really being much of a programmer.

You can get a BSEE without being much of an engineer, too. BSEE new grads definitely get tested by hiring companies to make sure they know their shit, but it is nothing like what this industry does.

Last one I had was to implement QuickSort. Had a -huge- argument with the interviewer who insisted there was an off-by-one error and it would never work. Needless to say, he was thoroughly wrong once it was coded up and I didn't get the job. :)

> It was such a turn off

It's pretty much an automatic "no" for me now too.

After the contract was over, I was able to negotiate a higher pay to convert to perm than the job I declined offered even with the signing bonus.

In my experience, the best type of coding tests are based on creating a skeleton program with failing unit tests already written and pair program with the candidate and see how they code and make the unit tests pass.

Currently I have two phases in my unit test based interview.

Phase 1: simpler tests and requirements that they have to make pass.

Phase 2: more complicated requirements and a second set of unit tests they have to make pass without breaking the first set.

To me, that mirrors how we actually work - get an MVP out the door relatively fast and add features later on ensuring you don't get regressions.

I've (thankfully), never had to whiteboard. Think it's a US/SF thing.

That these interview techniques are determined so much by culture, fashion, and following what the successful companies do (I.e. Google), in itself shows how useless they are.

That said, I acknowledge I can't think of a perfect way to assess software engineers. We have a take home exercise, but ultimately gut feel drives the decision.

And that has spread all over the world now, because a lot of companies outside of SV, tend to view these BigWebCo's as something to emulate and imitate from, and are doing these whiteboard interview all the time.

In fact the whole point of these Hacker rank kind of sites is to basically provide the web equivalent of a white board. You are asked to submit code(Typically some question based on a exaggerated edge case of some algorithm) and see if it passes all test cases. You are in if it does, out otherwise.

The fact that some companies use bad practises for hiring doesn't change how hard it is to judge someone's code from their Github repo. I agree that whiteboard tests are horrible, but switching to a different bad idea wouldn't help.

Unless you're looking at a toy or unfinished project, most code that's released (on npm, maven, or something similar) is actually fairly high quality. Certainly higher quality than what I've had to work with in production systems that were making millions of dollars a month.

You're moving the goal posts from "Has code in public" to "Launched, contributed to, or otherwise involved with a successful open project." That's quite different, and arguably a lot of hiring managers do look at that sort of thing very favourably.

I think this is complete BS. There's people much smarter than me that are getting screwed by these terrible practices. Max Howell, Homebrew's author, was famously rejected by Google for not passing some stupid whiteboard exam, when just about everyone at Google actually uses the software he authored.

I'll defer to Max's own explanation for that: https://www.quora.com/Whats-the-logic-behind-Google-rejectin...

As he says himself, Homebrew is a successful product but not necessarily great computer science. He 'failed' a computer science question. We can question whether or not Google needed to ask that in the interview, but they clearly thought it was important enough for the role Max was applying for to ask it. So maybe rejecting him for that role was the right decision. We don't really know. We do know that he wasn't applying to Google for the job of "Homebrew Maintainer" so his work on Homebrew, while really impressive, isn't really the whole story in hiring him. Surely you're not suggesting that the author of a successful project should be able to walk in to any developer role regardless of what's needed to do the job?

Maybe Google is different but I usually need to do computer cine type stuff about one a year.I have used recursion on maybe 4 different occasions in my carreer, but that's the sort of thing that comes up regularly in coding tests. Why not test for things that are actually being used on the job?

A public code repo is more than just code.

* It's documentation

Is there a decent README or other background info on how to get setup? Sample data? Tests?

* It's activity

Was this a fork with one patch and nothing since then? Or is it a long-term project with regular/periodic activity?

* It's community

Is this project getting bug reports, feature requests? How is the owner dealing with them? Ignoring them? Being rude? Actively engaging? Delegating? Honestly admitting they don't have the time and asking for help?

There's a whole mess you can judge about someone that goes beyond "code" when looking at a repo. I know someone who was hired at a company a couple years back who was brought in because of a friendship but the "github" profile was referenced as justification. "This person has a lot of projects and stars - they're definitely active!". The person had starred other projects, and had cloned about 40 repos, and made (trivial) changes in about 5 of them. They had no original code. The person had been brought in to lead a team of about 6 developers, and left within 2 months because they had no experience working with actual production code or working with people on a team. That was quite apparent from their github profile.

Let me take that back. I got my first development job in college from another school about 100 miles away because he saw a HyperCard Eliza clone I wrote and posted on AOL back in 1993.

I recommended hiring him based on a github profile.

I'm now CTO of a company operating a space that's as challenging as it is esoteric. We make hardware for adding by-wire controls to some existing cars to do autonomous vehicle test/validation. We make high-fidelity, high-bandwidth data capture software for AV sensor mules. We're working on a SDLC that enforces cryptographic chain of custody and produces safety-certification evidence as a continuous delivery artifact, and we're working on a high-assurance runtime environment for AV applications.

Hiring for this diversity of internal & external products would appear to be extremely challenging, but reforming/growing the existing teams and building new teams has proven fairly easy. Insofar as quality of new hires, productivity & growth of realigned staff, etc. is concerned. The challenge was in all the preparation it took to make it easy.

1. I never hire until I am absolutely certain I can identify and articulate a very specific need that aligns with a very specific set of goals.

2. I never hire until I am absolutely certain I understand the responsibilities and level of ownership required to meet those goals.

3. I never hire until I have a meaningful frontier set of models for the personality profiles (fallabilty acknowledged) that are likely to align with both those responsibilities and of the people they'll work most closely with.

4. I have candidates pair with someone or someones they'd actually work with on an actual problem being actively worked on by the team. Why contrive a pointless artifice like programmer-trivia---to guess at what the person would be like on the team---when I can make them an honorary member of their actual future team for a couple hours on-site or over video/screenshare?

5. I don't try to get a bargain on my hires. If I've managed to do all the above, then I know what I need, I know what it's worth, and I know how I need to manage my financial constraints.

This is my problem with things like HackerRank. It's just a thing that hiring managers can hide behind as a kind of faux-rigor to produce a completely meaningless "score" as though such a complex thing can be distilled down into a single number and used as some common metric for comparison.

Hiring is too important to the company, the team, and ultimately the candidate---whose livelihood you now partially hold in your hand---to be so fundamentally lazy about it.

We have been working super hard (and will continue to) to emphasis on 3 aspects -- problem solving to test computational thinking with basic data structures, role-specific knowledge (for e.g. react if you are hiring a front-end developer) & language proficiency (strength in at least one language) as the 3 pillars.

Given the foundation of HackerRank & the overall associated impression of algorithm challenges, we often get bucketed into a "puzzle company". Obviously, this is on us to make the narrative better and we are working towards it.

Ultimately, the goal of us is to ensure developer experience is great -- showcase your skills in the language/framework you are comfortable in with more time (not a 60-min test). It will take sometime but we are heading in that direction.

This happens more often than you'd think.

My previous employer eventually made HackerRank a requirement for hiring. Fortunately, by that time I had already basically figured out how to freeze the size of my team despite having an absurdly large headcount projection attempting to being forced onto it, so I was able to ultimately abstain from using it (after catching some hell for actively opposing it).

However, if I had been in a position to have to use such a tool, there was a major failing in the way it worked (at least 2+ years ago) that would have made it impossible to derive useful meaning for me from the results.

What was critical to partially assess for me was a person's ability to fully understand the nature of basic assumptions they take for granted. To get a sense of their completeness of thought and appropriate self-doubt in that completeness.

I wanted to do this by being able to provide a black-box implementation of two simple functions:

add/2 multiply/2

Both of which would be intentionally implemented incorrectly. Then I wanted to be able to have the user write out property-based tests that would help them discover the way in which the opaque underlying implementation was flawed despite the fact that they would have no internal knowledge of the implementation.

This would have very roughly allowed me to assess:

1. How they dismantle basic lifelong concepts that have become mental reflexes like addition and multiplication, but in terms of their properties and invariants (associativity, commutativity, identity, etc).

2. Then also whether or not they could infer what was wrong with the simple opaque implementation based on which properties/invariants weren't holding as they should have. So that I could see how they handled their confidence in the completeness of their analysis.

No idea if you could do this today, but at the time the HackerRank system couldn't usefully support the above use case. Or at least the person who conducted the training had no idea how one would do such a thing.

This is the absolutely best skills test I've seen so far. More companies need to try it out.

One of the biggest benefits is that it removes the adversarial relationship of a normal whiteboard interview. That kind of adversarial relationship almost never happens day to day (and if it does something is very wrong).

There's a tremendous amount of variability in whether or not the artificial test that's been created actually maps appropriately to what you're trying to assess. It's extremely difficult to do well. There's further variability in the proctoring and posturing in how the assessment is performed, not just the assessment itself.

Almost certainly you've added layers of obfuscation to the thing you actually want to know, unless what you want to know is how good the person is at taking your specific test, which may or may not usefully test anything, and which may or may not be administered in a way that could get useful results even if the test itself is meticulously well-crafted.

If you're intent on, or can't avoid, operating a more traditional looking hiring process, then I'd suggest at least trying this. Even if the assessment is contrived, then use questions that nobody in the room knows the answers to, including yourself. It doesn't relate to the work directly necessarily, but it can at least help mitigate the effects of survivorship bias.

And none of those interview questions will measure how the candidate performs in your team's day to day environment, nearly as well.

Mostly it's a framework for how to think about dimensions of the problem in the abstract. It's like "team architecture" analysis. I use the OCEAN trait model. I'm aware of it's limitations (and they are numerous), as well as those of other models and methods. The reason I use it is because it's a simple enough, but still rich enough, model for me to be able to partially apply on-the-fly through casual conversation, etc.

For me it breaks down roughly like:

1. I think about that job that I need done and plan traits that probably map to that job. The kind of expressed traits that surface in a fastidious-finisher are very rarely also the traits that surface in a cowboy-hacker (the coarseness of the caricatures is for illustration only).So there's a sketch that looks like: O-low, C-mid, E-NA, A-mid, N-low or something for any given need.

2. I plan out the set of traits that are mandatory to be expressed (or not expressed) strongly. Like it might be the case that I need someone to be super high on C, but if they're also super high on N, and being high on N is fairly undesirable for the current situation, then that warrants further exploration. The point of this is to try to help prioritize focus on what to assess, how to assess it, reasonable confidence in the assessment, and which dimensions are fine to be really fuzzy.

3. With model priorities in place, when speaking to people there are a lot of ways to prompt, leave empty airspace, and methods of story-telling that facilitate creating safer space for a broader spectrum of interaction than is often expected in a normal hirer-hiree dynamic. These help guide filling in that rudimentary sketch. Talking about almost anything other than the job itself helps with this because it's important to try to establish a peer-like posture. With a lot of practice one can get "okay" at doing this in such a way that it can be a meaningful, but incomplete, weight in the calculus on how to proceed with a candidate as well as the team the candidate would be joining.

4. Lastly, the most important thing to do here from my experience is to bias myself toward constantly trying to rapidly invalidate my model sketch at every possible stage of interaction from the first email exchange all the way up to issuing an offer. This does a few things. It forces me to have constant and active intention in every single step of the process and every single interaction with a person. It keeps me from religiously believing in the modelling itself because it's constantly clear that it's incomplete and fallible. Perhaps most important is that it puts the adversarial focus on myself and my assessment process rather than laying that adversarial relationship on the candidate.

I have to read as much or more about psychology and neuroscience as I do about byzantine fault-detection or applications of formal methods. Hiring can't be an afterthought that we as hiring-managers take lightly or try to relegate to passive tools or industry norms that seem patently ridiculous on their face.

Every "bad hire", every misaligned/unsatisfied employee, and every termination is ultimately the fault of management.

I was thinking, since public repositories or open source contributions are the new "mandatory". If you want to see me coding, just request a new feature to one of my public repositories on github. These coding assignment you sent by email? I don't even bother to read them, in fact I store them scrupulously together with your email - once you might be looking for a job and I will be the one doing hiring.