News for hackers. Stuff that matters.
Re:Show HN: Real world (Jupyter notebook embed) way to asses... by shikharja on Thursday August 29, 2019 @02:07PM (#20832082)
ska, our aim with the challenge was to allow candidates to not be biased by a fixed outcome and try to solve the problem as they would solve any real data science problem.

This meant we couldn't automatically score/rank a candidate's solution. We do provide them with an evaluation metric in the problem description (Mean Absolute Error). Here is scoring rubric we provide to the interviewers when they review the submission - https://d.pr/i/hNYY0u

Would love to hear more opinions on our scoring rubric

Re:Show HN: Real world (Jupyter notebook embed) way to asses... by ska on Thursday August 29, 2019 @02:13PM (#20832125)
That's useful information, thanks. I'll add some thoughts on the rubric.

[edit]. Initial thoughts:

- "data wrangling" scoring difficult given this task - more weight to "rationale", that's more important the "performance", here.

- not enough focus on communication capabilities

- really need something on validation

- "proficiency" measure you use is pretty much impossible to accurately evaluate from your example question

- way too much weight to modeling section overall

Re:Show HN: Real world (Jupyter notebook embed) way to asses... by shikharja on Thursday August 29, 2019 @07:04PM (#20834555)
By 3rd bullet, you mean Data Validation?
Re:Show HN: Real world (Jupyter notebook embed) way to asses... by ska on Friday August 30, 2019 @07:58PM (#20843810)
both data validation and model validation are important.

Of course more abstractly you want to validate the entire process - but I was referring to those two, as it is hard to see how to address the latter in a format like this.