Have you done this? My worry would be that sooner or later you'll have a candidate who posts the assignment online ruining your weeks of effort. It also seems like it makes it much easier to cheat if you make it a take home.

Yes, I ran a process like this for several years. I ran recruiting for the largest software security firm in the US; before that firm bought us, I used this process to more than double the size of my company. When I left, to work on a recruiting startup, not one of the people I'd overseen hiring had quit or been fired.

I'm religious about this topic for a reason. :)

Was the 0% turnover due to great teammates, great projects, or above-market comp? I've never seen 0%!

We paid the market median to new hires; we definitely didn't buy our way to that turnover (NCC pays better than a lot of early stage startups, but not better than late-stage ones).

No, I attribute the turnover to the recruiting mechanism. We found great people who were sorely mispriced by the market, and we took advantage of that to create a win-win scenario: people without the resume to get a similar job at a competing firm got an extremely impactful resume bullet and a good-paying job, and we got people who genuinely wanted to be on our team and weren't applying as a once-a-year lateral job-hopping gamble.

(I have no problem with people job-hopping, by the way, but every employer is trying to minimize their exposure to that.)

You were also recruiting for a "cool" sector where people would be learning marketable skills. Do you have any evidence that the same technique would work for more "boring" companies? (I ask because I'd like to convince my manager that creating a work-sample test wouldn't be a waste of time)

I agree completely. But in many organizations recruiting and hiring would be considered overhead instead of strategic, and thus a target for outsourcing to someone in HR or some outside firm. The problem there of course is that those people have neither the ability nor the inclination to create or evaluate a work sample challenge.

Really my comment was just to help demystify why this sort of thing doesn't happen in many companies based on my own observations.