Instead, we're going to just expose raw data about somebody's code submission, and let you decide what those numbers mean.
Some examples:
1. 7/10 unit tests passed
2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB
3. Linter returned 10 warnings, 0 errors
4. Cyclomatic complexity counter returned max of 24
5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars
In my experience, the metrics you've sampled aren't particularly useful for hiring. Some of my thoughts:
> 1. 7/10 unit tests passed
Automated unit tests are table stakes on all platforms like HackerRank, LeetCode, etc. Might be useful to classify/clarify the failing ones? Or even add an option for the candidate to clarify their thought process for those.
> 2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB
As an engineer, I see the appeal of OOM checking. But as a candidate, it would be important to have the same information available during test-time, so I could fix it. Could be as simple as an accidental N+1 ORM mistake. Otherwise, hidden performance testing would make the experience very stressful, as then I need to double-check everything and over-optimise for perf.
> 3. Linter returned 10 warnings, 0 errors
I would argue that linting should be out-of-the-box and automated for the candidate. Or even if not, there should be heavy filtering of linter warnings. Every company has their own style guides, and candidates shouldn't be expected to learn that for an interview.
> 4. Cyclomatic complexity counter returned max of 24
While I do understand the concept of Cyclomatic complexity, I have /never/ seen this used IRL in any company (startup, mid-size, FAANG) that I've worked at. I would much rather judge code complexity in terms of readability and decision making.
> 5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars
I appreciate clean git commits, but these are enforced by company's internal policies during code reviews. Easy to learn once, and continue doing. But definitely not a deal-breaker in a time-constrained interview assignment.
Are these raw metrics you shared what constituted the grading from V1? I would definitely reevaluate most of these for V2, as they seem a little nitpicky than insightful.
I've interviewed hundreds of engineers in the last twenty years. I've been (and am) a hiring manager for many of them. I wouldn't touch this with a ten foot pole, and I'd run away as a candidate if my git commits were being judged as part of a coding interview.
These "metrics" are for conversation, not for automated "judgement."
My question is around the grading examples. How do you automatically determine Functionality is Senior - Weak or Commit History is Intermediate - Strong?
I find these to be quite subjective and better understood as a whole for any given candidate. The grading mechanism provides a false sense of being unbiased, and further dehumanizes the process.
> Every metric we measure is numeric, and every test is deterministic. As a result, you can make hiring decisions based purely on merit.
I fully disagree with this for hiring, unless you’re looking for machines, not humans. Measuring things automatically does not result in purity of bias.