None of them actually tried solving it within the constrains that a candidate is put through.

Larger projects take up more time and introduce a larger possibility that some matters of opinion or taste will impact the candidate's performance. I don't believe differences in taste are important as long as the candidate demonstrates that he is able to comply with a defined style guide.

The actual test is going to depend on the position at hand, but one of my favorites is a very simple program that asks the candidate to use GitHub's API to display the list of public repositories under a user-inputted username. That's it. I tell them they can use any language they want.

This simple test gives all the information we need about the basics:

a) the candidate is able to go online, provision himself an API key, find docs, and reference those docs to see an external vendor's API format

b) the candidate is able to use that information to craft a program that successfully interacts with the vendor's endpoint

c) the candidate is able to present the information in a concise, desirable manner.

d) the candidate is able to do all of this with a simple 1 paragraph description of the project.

The choices the candidate makes in the process of completing this simple task tell you a lot about his process, style, habits, and preferences, even though the project is very minimal in its actual requirements.

I've had people give me web apps, command-line apps, and GUI apps that accomplish this same goal. Many candidates would go above the requested specifications and many candidates would reply the same day they were given the test, which to me was an excellent signal that they felt their time was respected and that we were doing a good job of engaging them and making them interested in working for us.

As I stated, this test is not appropriate for all positions, but I think most tests should be modeled after those principles. Give the candidate room to express himself and demonstrate relevant practical knowledge.

I hope you'll consider a minimalist project like this over something like "design a multiplayer game ... in 3 hours".

Have you considered requiring a thought process journal as well?

In terms of project choices, to me, it seems like you guys erred on the side of choosing projects developers would find interesting and challenging technically over projects that are practical and accurately represent the kinds of work most developers will actually be hired to do. I'm not going to post the details here for obvious reasons, but out of the four projects offered, only the multiplayer game was even remotely relevant to front-end/back-end or even application development in general, which are areas that probably account for the vast majority of development work available from startups. I realize there needs to be a balance to be stuck here, but in my humble opinion, as a recruiting firm, you should be erring on other, more the practical side in terms of project choice. I applied to Triplebyte to find a job, not to fulfill my intellectual curiosity (I can do that better on my own time without needing someone to assign projects to me).

In terms of project scope, I'm not really qualified to comment on the other choices, because they were way outside my area of expertise, but the multiplayer game definitely didn't feel like a 3 hour project. As suggested in another reply, I really hope you guys can actually give the project a try yourself and see what level of completion can reasonably be expected from three hours of work on something like that. Take whatever times reported by candidates who have successfully completed the project with a grain of salt, because people will have a tendency to understate the level of effort they spent to make themselves look more efficient (no matter how much you tell them you don't care). Here's another possible idea for making projects that take reasonable amounts of time to complete: just take your traditional interview questions and slightly extend them a bit with extra features, and simply expect better polish, architecture, test-coverage, and overall code quality, etc during the code review.

Anyways, my experience with Triplebyte's take-home interviews definitely didn't leave a great impression, but I still recommend you guys to my friends and colleagues because I do want to support what you guys are trying to do. Hopefully you can take some time to revisit some of the issues people have mentioned and make the necessary improvements. I'm happily employed now, but I'd love to give Triplebyte another try the next time I'm looking for work. =)