I have to disagree with you on the "more than one interviewer" point. If you are only ever going to have your abilities assessed by one person, then you are 100% at the mercy of that one person's biases.

A single interviewer will almost always hire someone who is exactly like him or herself. That's pretty much the opposite of what you want to do if you want to increase diversity.

You're absolutely right that you shouldn't have a big panel, but two interviews with two people each shouldn't be overwhelming while still getting diversity in your hiring. If one person says "terrible x" and another person says "good x", then there's room for discussion about what particular things were good or bad, and on balance, whether the benefit of having that person is worth the risk.