The short story is that hand-curation does not seem to work at a very large scale, so we need to create tools for automated discovery (e.g., search, recommendations, controlled skill vocabulary etc.) and automated but credible signals of ability and trustworthiness (e.g., feedback system, skills tests, verified identities etc.). In a nutshell, the challenge is trying to take the things Josh was doing by hand and turn them into data-driven features.

Note: I'm the staff economist at oDesk & I'm on the research/data science team.