I agree that the kind where both parties come together and agree, for a specific case, to engage in arbitration and be bound by the result is OK. The kind where you agree beforehand, when no damages have occurred, that if any case comes up then you will both go to arbitration instead of court, is not OK.
Look at the NFL and the players’ association. They have an arbitrator, the very capable Professor Stephen Burbank, handling disputes. Both sides (and the courts) strongly prefer this to having to go to court.
Arbitration is more controversial when foisted upon consumers in contracts of adhesion, i.e. the take-it-or-leave-it contracts you sign all day at stores, amusement parks, websites, etc. Also very controversial is that arbitration clauses are now being used to prevent class action litigation altogether. In those cases, the Supreme Court has nevertheless upheld it through a very broad reading of the Federal Arbitration Act.
Another way to state the same thing: very few people enter a contract contemplating that they will be the one breaching it, so they are willing to agree to a system for efficiently adjudicating conflicts. But circumstances change and make breach attractive for a particular party, and at that point they will want to protect their advantage, fairness to the other party be damned.
I think you're justified in having concerns about arbitration, but I don't think the line that you're drawing is the right one. Focusing on the nature of the relationship between the parties might be a more fruitful approach. I don't see any reason why two sophisticated entities, truly engaged in back-and-forth negotiations where each has leverage, should not be allowed to agree to binding arbitration between them for future disputes.
For broad definitions of 'people', I don't think this is true. I am utterly certain that businesses enter into these contracts secure in the knowledge that it is much more likely that wrongs will be alleged against them under the contract than that they will need to allege wrongs against the contractee, and that businesses feel that arbitration is likely to favour them. If businesses didn't believe at least the latter, or even if there were just divided opinions, then arbitration clauses would not be included in every. Damn. Contract.
I think this is the key point. If there's a power disparity (commonly, adhesion contracts), the favored party should not get to pick the arbiter.
Take the "court room" TV shows like Judge Judy, they are an example of binding arbitration that both parties benefit from. The producers look for cases pending in small claims court and offer them to drop their case and settle their dispute in arbitration instead, that arbitration is what's shown on TV. The show pays all parties an appearance fee plus travel expenses and the show pays any judgement that is awarded instead of the defendant. Since it's not a judgement it doesn't show up on public records so it wouldn't affect the defendant's credit report or public records.
Being forced into binding consumer arbitration is certainly terrible and ought to be illegal. Arbitration should not be forced when one party has significant less power than the other. Especially when only one party gets to pick the arbitrator.