What I mean is that, I put together a few challenges with APIs we work with (or gems we've written, that sort of thing) and have them solve a problem that will require them to use their innate ability, drive, and Google/StackOverflow/etc. to solve the problem.
It's not something that they would be able to do without the internet. Does that make sense?
The contract thing is tough, although at least in NYC the market is so tight that there's really no risk of unemployment, but yeah, we wouldn't do relo + contract. We'd start them off as remote and see if that worked. If so, then relo; it's risky risky for both parties (especially in NYC) that it's kinda crazy to expect either party to agree to it.
The candidate has no idea what NYC is like, and the organization may be reluctant to fire a bad hire who just isn't fitting in because they feel bad. At least with a remote contract, no one has to move and we can see if we both like each other enough to get married. Then would come the offer + relo., or if it's not a good fit, end the contract (with 2 weeks notice at least).
Finally, I'd say the initial screen is a bit light. We used to have about a 1-2 hr. technical Q&A followed by Fizzbuzz-style questions, but that had a low false-positive rate so we got rid of it.
That is to say, the background work I did (googling them, stack overflow, github, twitter, their resume, etc.) was a pretty good filter at keeping out the obviously underqualified.
However, I'm considering putting it back as it helped give me a more nuanced picture of the candidates that I end up doing as part of the contract, and it might help to have some first-impressions ahead of time (but I am always worried about false first impressions clouding my judgement).
To your process, my thoughts:
- Open-book challenges like you describe (in which you can use stack overflow and whatnot) are awesome. However, why do you have them do stuff they're not expected to know? From what I've read, the only thing hiring studies seem to agree on is that the questions which best predict a candidate's long-term success are those that simulate what the candidate will actually be working on. Getting them to learn outside their comfort zone is good, but maybe you'd be better off doing that by finding problems you've actually had in the course of your work that are outside of their comfort zone.
- Short-term contracts are tricky. On one hand, I think they're the best way to test how well a candidate will work out. On the other hand, switching jobs is a lot of work and stress. If I had a choice between two pretty good companies, one of which was a 1-month contract and one of which was a normal job, I'd definitely take the latter. If I had to move to get your job, it'd make rejecting such a proposal even more of a no-brainer. If it works out for you, that's great, but I'd be really worried about losing excellent candidates.
- It seems like your remote screening is lighter than what I'm used to. At my current gig, we send candidates a challenge to be completed in their own time (3-9 hours of work)- we've found it's a great prelude to the phone screen with an engineer. It's pretty much a code sample for which we have an excellent reference (all the previous submissions). We've found it improves the quality of candidates that get to our on-site.