We have been working super hard (and will continue to) to emphasis on 3 aspects -- problem solving to test computational thinking with basic data structures, role-specific knowledge (for e.g. react if you are hiring a front-end developer) & language proficiency (strength in at least one language) as the 3 pillars.
Given the foundation of HackerRank & the overall associated impression of algorithm challenges, we often get bucketed into a "puzzle company". Obviously, this is on us to make the narrative better and we are working towards it.
Ultimately, the goal of us is to ensure developer experience is great -- showcase your skills in the language/framework you are comfortable in with more time (not a 60-min test). It will take sometime but we are heading in that direction.
This happens more often than you'd think.
My previous employer eventually made HackerRank a requirement for hiring. Fortunately, by that time I had already basically figured out how to freeze the size of my team despite having an absurdly large headcount projection attempting to being forced onto it, so I was able to ultimately abstain from using it (after catching some hell for actively opposing it).
However, if I had been in a position to have to use such a tool, there was a major failing in the way it worked (at least 2+ years ago) that would have made it impossible to derive useful meaning for me from the results.
What was critical to partially assess for me was a person's ability to fully understand the nature of basic assumptions they take for granted. To get a sense of their completeness of thought and appropriate self-doubt in that completeness.
I wanted to do this by being able to provide a black-box implementation of two simple functions:
add/2 multiply/2
Both of which would be intentionally implemented incorrectly. Then I wanted to be able to have the user write out property-based tests that would help them discover the way in which the opaque underlying implementation was flawed despite the fact that they would have no internal knowledge of the implementation.
This would have very roughly allowed me to assess:
1. How they dismantle basic lifelong concepts that have become mental reflexes like addition and multiplication, but in terms of their properties and invariants (associativity, commutativity, identity, etc).
2. Then also whether or not they could infer what was wrong with the simple opaque implementation based on which properties/invariants weren't holding as they should have. So that I could see how they handled their confidence in the completeness of their analysis.
No idea if you could do this today, but at the time the HackerRank system couldn't usefully support the above use case. Or at least the person who conducted the training had no idea how one would do such a thing.
This is the absolutely best skills test I've seen so far. More companies need to try it out.
One of the biggest benefits is that it removes the adversarial relationship of a normal whiteboard interview. That kind of adversarial relationship almost never happens day to day (and if it does something is very wrong).
There's a tremendous amount of variability in whether or not the artificial test that's been created actually maps appropriately to what you're trying to assess. It's extremely difficult to do well. There's further variability in the proctoring and posturing in how the assessment is performed, not just the assessment itself.
Almost certainly you've added layers of obfuscation to the thing you actually want to know, unless what you want to know is how good the person is at taking your specific test, which may or may not usefully test anything, and which may or may not be administered in a way that could get useful results even if the test itself is meticulously well-crafted.
If you're intent on, or can't avoid, operating a more traditional looking hiring process, then I'd suggest at least trying this. Even if the assessment is contrived, then use questions that nobody in the room knows the answers to, including yourself. It doesn't relate to the work directly necessarily, but it can at least help mitigate the effects of survivorship bias.
And none of those interview questions will measure how the candidate performs in your team's day to day environment, nearly as well.
Mostly it's a framework for how to think about dimensions of the problem in the abstract. It's like "team architecture" analysis. I use the OCEAN trait model. I'm aware of it's limitations (and they are numerous), as well as those of other models and methods. The reason I use it is because it's a simple enough, but still rich enough, model for me to be able to partially apply on-the-fly through casual conversation, etc.
For me it breaks down roughly like:
1. I think about that job that I need done and plan traits that probably map to that job. The kind of expressed traits that surface in a fastidious-finisher are very rarely also the traits that surface in a cowboy-hacker (the coarseness of the caricatures is for illustration only).So there's a sketch that looks like: O-low, C-mid, E-NA, A-mid, N-low or something for any given need.
2. I plan out the set of traits that are mandatory to be expressed (or not expressed) strongly. Like it might be the case that I need someone to be super high on C, but if they're also super high on N, and being high on N is fairly undesirable for the current situation, then that warrants further exploration. The point of this is to try to help prioritize focus on what to assess, how to assess it, reasonable confidence in the assessment, and which dimensions are fine to be really fuzzy.
3. With model priorities in place, when speaking to people there are a lot of ways to prompt, leave empty airspace, and methods of story-telling that facilitate creating safer space for a broader spectrum of interaction than is often expected in a normal hirer-hiree dynamic. These help guide filling in that rudimentary sketch. Talking about almost anything other than the job itself helps with this because it's important to try to establish a peer-like posture. With a lot of practice one can get "okay" at doing this in such a way that it can be a meaningful, but incomplete, weight in the calculus on how to proceed with a candidate as well as the team the candidate would be joining.
4. Lastly, the most important thing to do here from my experience is to bias myself toward constantly trying to rapidly invalidate my model sketch at every possible stage of interaction from the first email exchange all the way up to issuing an offer. This does a few things. It forces me to have constant and active intention in every single step of the process and every single interaction with a person. It keeps me from religiously believing in the modelling itself because it's constantly clear that it's incomplete and fallible. Perhaps most important is that it puts the adversarial focus on myself and my assessment process rather than laying that adversarial relationship on the candidate.
I have to read as much or more about psychology and neuroscience as I do about byzantine fault-detection or applications of formal methods. Hiring can't be an afterthought that we as hiring-managers take lightly or try to relegate to passive tools or industry norms that seem patently ridiculous on their face.
Every "bad hire", every misaligned/unsatisfied employee, and every termination is ultimately the fault of management.
I'm now CTO of a company operating a space that's as challenging as it is esoteric. We make hardware for adding by-wire controls to some existing cars to do autonomous vehicle test/validation. We make high-fidelity, high-bandwidth data capture software for AV sensor mules. We're working on a SDLC that enforces cryptographic chain of custody and produces safety-certification evidence as a continuous delivery artifact, and we're working on a high-assurance runtime environment for AV applications.
Hiring for this diversity of internal & external products would appear to be extremely challenging, but reforming/growing the existing teams and building new teams has proven fairly easy. Insofar as quality of new hires, productivity & growth of realigned staff, etc. is concerned. The challenge was in all the preparation it took to make it easy.
1. I never hire until I am absolutely certain I can identify and articulate a very specific need that aligns with a very specific set of goals.
2. I never hire until I am absolutely certain I understand the responsibilities and level of ownership required to meet those goals.
3. I never hire until I have a meaningful frontier set of models for the personality profiles (fallabilty acknowledged) that are likely to align with both those responsibilities and of the people they'll work most closely with.
4. I have candidates pair with someone or someones they'd actually work with on an actual problem being actively worked on by the team. Why contrive a pointless artifice like programmer-trivia---to guess at what the person would be like on the team---when I can make them an honorary member of their actual future team for a couple hours on-site or over video/screenshare?
5. I don't try to get a bargain on my hires. If I've managed to do all the above, then I know what I need, I know what it's worth, and I know how I need to manage my financial constraints.
This is my problem with things like HackerRank. It's just a thing that hiring managers can hide behind as a kind of faux-rigor to produce a completely meaningless "score" as though such a complex thing can be distilled down into a single number and used as some common metric for comparison.
Hiring is too important to the company, the team, and ultimately the candidate---whose livelihood you now partially hold in your hand---to be so fundamentally lazy about it.