"ssh -i /path/to/key" seems like it would do what you want, which makes me think I don't understand the problem you're trying to solve.