Why not just give them different parts. The last guy you interview you give the task of putting together the different pieces, and then you have your finished application. Of course you should also interview QA people so they can find the problems with what they other guys did...

Assuming you're not joking...

Aside from that just being a very dishonest way to conduct business, you'd spend all your time organizing people who are working together who don't know they are working together. Even if you could manage that, you'd have no way of assembling the finished components into a working application, and there is just no way you could find someone capable of handling that task that was also foolish enough to think such a complex operation would be part of a job screening process conducted by a person incapable of knowing if it's done well or not. In fact, it's very unlikely you'd fool any experienced developers they'd quickly deduce that you don't know enough about the code to judge it's quality.

The end result would be a pile of code that wouldn't work and you'd have no way of fixing it without hiring an even more exceptional programmer, who would probably tell you to throw out 90% of the work and start over.

Well, you are unlikely to get a finished application in two man weeks. You have everyone implement the same thing so you can compare apples to apples. Then you hire the best for the long term.

Assuming that works and you actually get a few decent candidates to submit solutions, how exactly do you propose that the business guy/gal selects the winner? He can't evaluate their code, he can't evaluate their design decisions... What criteria exactly do you think the business guy/gal would use to select the winner?

You would have to select a project that produces output the business user can judge. So if it's a UI, the business user can make sure all the parts are functional, the load times speedy, the edge cases handled, etc.

Defining edge cases without technical knowledge is problematic, load times are meaningless without load testing, so basically all you'll know is if the code works, and that is a very low bar to set especially for a startup.

And even if you could do all these things without technical knowledge at best you'll be able to weed out the mediocre, never find the best.

Yep, that's just what I was thinking. You could even have existing members of your team implement it, to make sure you weren't diluting them by hiring worse programmers.