I struggled with this when I was responsible for hiring at 500px before I moved on. The key is to ask for code samples ahead of time. It doesn't necessarily have to be open source contributions, though that slightly preferred, but I refused to interview someone that didn't have code samples.

I went from 30 interviews per one hire to about 2 or 3. The most common problem being a potential candidate not knowing enough of the internet stack to be truly useful (for example, knowing only rails, but having no real knowledge of how AJAX worked, or load balancers, or databases, or DNS, etc).