> Personally I think it'd be fine to e-mail back and say "sorry I don't have time to work on this with my current schedule, could I do an in-person interview instead?"

What happens instead is that you just don't ever hear back from candidate.