I've been on the reviewer side of a handful of "choose your own language"-style take-homes recently and found that it's really not good for the candidate if they actually end up using something that wouldn't have been the interview committee's first or second choice. There have been cases where choosing a less-trendy-but-still-totally-viable toolkit has effectively disqualified a candidate, with several committee members not even considering it necessary to look at the code. This is very unfair and lame but an unfortunate reality. I asked that the test be changed to constrain the options at least to a list that wouldn't be immediately disqualifying.
You could advise the candidate ahead of the interview something along the lines of "you'll be asked to write a code sample in either Ruby or Python -- you'll have full access to Google, but you may want to brush up on the basics of these languages if you haven't used them recently".
This does two things: first, it prevents the issue you have, where you're essentially not sure if you're correctly scoring the samples produced. Secondly, it really requires you to constrain the problem to things that someone who has barely used language X or Y can do within 45 minutes.
Ah, but there's the rub. Candidates are trying to please the interview panel. If you don't provide guidance, the odds that they'll just use whatever they think the interviewers most prefer are just as good, if not better, than the odds that they'll actually use whatever makes them most comfortable.
You said yourself that since some candidates pick a language that you don't know well, you can't really tell if the failure of a large number of those candidates is reflective of a bad test or just a mismatched candidate pool. IMO, if you're going to stick to the "pick any language" thing, you should at least find out and ensure that any language the candidate picks will have a fair shot.
> it can even be a better interview because we get more insight on how the candidate communicates and their ability to walk someone else through their solution.
You can still get the candidate to communicate and explain his choices if you give an option: "either Ruby or Python" or "either JavaScript or Visual Basic", etc. The problem with having this happen in a language that the interviewers don't know reasonably well is that they are much more vulnerable to the smooth talker who can present incorrect information confidently, and they won't have enough background/anchoring in the subject matter to know the difference.
> A potential other bonus is less biases leak through from an interviewer on "that is a strange way to do that in language X."
I would say that if you're worried that interviewers will load in biases toward their preferred shortcuts etc in a specific language, that you should be equally worried that some good candidates are being excluded for choosing the "wrong" language in an any-language-goes test.
Above, you mentioned that there'd be a positive response if a candidate used "Rust, Scala, or a lisp dialect" -- these are all relatively trendy. What if the candidate used nim, Pony, or some other language that hasn't pulled in to the hype superstation yet? What if the candidate used a language that's not-so-trendy anymore, like Visual Basic, Cobol, or bash? What if the candidate used a programming language of their own design, and brought a copy of the compiler with them on a flash drive?
I'm asking because I've seen this in practice. Candidates for a devops position who chose to use bash to implement the very simple take-home task they were given were laughed off by several other members of the interview committee, despite being potentially high-value senior people -- they were at least senior enough that they're more comfortable performing sysadmin-style tasks in a shell, rather than using a massive CM framework or an awkward amalgamation of Python scripts running os.spawn.
It feels like this type of thing happens a lot, in the same sense that very often, "unlimited PTO" just means "guess whatever amount of PTO is acceptable around here and hope you get it right".
I tend to give candidates a simple project already to go, with junit and hamcrest, possibly mockito already available, and ask them to go from there using a provided IDE (which I attempt to get set up as near as they prefer to work as I can). This generally works out fine. I certainly don't feel the boiler-platiness of the language gets in the way, mostly because the IDE is generally capable of doing most of the lifting with that respect anyway, but also because over the timescale of an interview question, we're generally only talking about a couple of classes at most.
For 20% of Java candidates, they do it just fine. Heck, a few echew the IDE and are fine working completely from the terminal (these tend to be particularly very solid at coding). Still wrestling with the idea.
The folks that already work with us (who wrote Java in a former role) see no problem with setting up a project nor do the folks that we hired recently (seeing as they likely passed that technical interview). But that all could be bias.
Maybe you are right. Maybe the next candidate or five in Java will get a base project and we can see how it goes.
I think Java depends much more heavily on powerful tooling to do the heavy lifting, and my experience of using that tooling when I haven't had a chance to configure it in advance has been pretty miserable.
If it's remote, I'll ask them to share their screen with me with a project set up and ready to go, having emailed them a copy of the interface we're going to implement about 30 minutes before the interview.
In general, if I'm asking them to code on my (or the company's) hardware, I'd start from an existing project, if only because I wouldn't expect them to be familiar with the installed tools (oh, you use maven? I'm a gradle user ... etc.) On their own laptop, I'd expect them to be more comfortable.
I'm doing a remote interview on Wednesday. That'll be on the candidates machine (because screen-sharing is easy, getting them inside the corporate network, not happening), and they've been told they'll need an IDE ready to go. I'll expect a project set up and ready to go before the call even starts.
What round is this?
Also, how much of their day-to-day work is going to be setting up new projects? I sometimes feel a better test is to be thrown into an existing code base and asked to make a change. It's far more indicative of the sort of work somebody is likely to actually be doing.
Yes, it does. And let me put it this way: I've used C# and Java at most of my jobs, those theoretically would be my comfort languages, yes?
I do not use those languages on interviews. I often just use C++ (!), or Rust (free unit tests!) if the company tools allow for it, or worst case I'd learn some Python basics. C#/Java are very awkward and boilerplatey in such a small time frame as 45 minutes.