I have a number of concerns about the efficacy of this, but they are made more difficult to rank by not understanding how you are planning to evaluate and use the results.

Can you elaborate?

Evaluation is subjective at the moment, by a review of the jupiter session by the hiring managers.

For certain data science usecases, evaluation is possible by using a CSV output bu a user, and comparing that to an expected CSV.

(I worked on the product).

Ok, although I would be wary of using a numerical comparison for anything except catching obvious errors.

I should have asked this before, but "data science" is a pretty broad term - who are you hoping to target with this? I'm guessing for pretty junior positions but want to clarify.

Oh, and one other question, do you /can you enforce the 60 min time? [edit: never mind, I answered this experimentally, you do cut it off at 60 min]

Agreed. Data Science is a very broad term. The challenge is designed for Data Scientists. We are trying to target all experience levels as of now through a screening/take-home test that should take about 60-90mins at a stretch. Do you think the timed challenge should be different for senior vs junior data scientists?

What skills would you consider important for senior vs junior Data Scientist?

I've added some top level comments.

For what it's worth, I think junior and senior DS roles should have fairly different evaluations & interviews.

I think at best this is fizzbuzz for DS, which is not inherently wrong. It's nice to know a software developer can write a loop and a data scientist can use a JN, so for weeding out people who have no practical experience with a given tool set, it could make sense.

The question then is how do you algorithmically (or even just consistently) distinguish a great data scientist from one who can accurately model answers to a question that was badly thought out?

Plus as pointed out before, the length of a take home could reduce applications from the most qualified candidates.

Wonder if this should be even shorter and more quiz/fun like so it intrigues rather than annoying more senior applicants, and still wondering the best way to identify the data scientists who ask better questions.

ska, our aim with the challenge was to allow candidates to not be biased by a fixed outcome and try to solve the problem as they would solve any real data science problem.

This meant we couldn't automatically score/rank a candidate's solution. We do provide them with an evaluation metric in the problem description (Mean Absolute Error). Here is scoring rubric we provide to the interviewers when they review the submission - https://d.pr/i/hNYY0u

Would love to hear more opinions on our scoring rubric

That's useful information, thanks. I'll add some thoughts on the rubric.

[edit]. Initial thoughts:

- "data wrangling" scoring difficult given this task - more weight to "rationale", that's more important the "performance", here.

- not enough focus on communication capabilities

- really need something on validation

- "proficiency" measure you use is pretty much impossible to accurately evaluate from your example question

- way too much weight to modeling section overall

By 3rd bullet, you mean Data Validation?

both data validation and model validation are important.

Of course more abstractly you want to validate the entire process - but I was referring to those two, as it is hard to see how to address the latter in a format like this.