Look at the NFL and the players’ association. They have an arbitrator, the very capable Professor Stephen Burbank, handling disputes. Both sides (and the courts) strongly prefer this to having to go to court.
Arbitration is more controversial when foisted upon consumers in contracts of adhesion, i.e. the take-it-or-leave-it contracts you sign all day at stores, amusement parks, websites, etc. Also very controversial is that arbitration clauses are now being used to prevent class action litigation altogether. In those cases, the Supreme Court has nevertheless upheld it through a very broad reading of the Federal Arbitration Act.
Another way to state the same thing: very few people enter a contract contemplating that they will be the one breaching it, so they are willing to agree to a system for efficiently adjudicating conflicts. But circumstances change and make breach attractive for a particular party, and at that point they will want to protect their advantage, fairness to the other party be damned.
I think you're justified in having concerns about arbitration, but I don't think the line that you're drawing is the right one. Focusing on the nature of the relationship between the parties might be a more fruitful approach. I don't see any reason why two sophisticated entities, truly engaged in back-and-forth negotiations where each has leverage, should not be allowed to agree to binding arbitration between them for future disputes.
For broad definitions of 'people', I don't think this is true. I am utterly certain that businesses enter into these contracts secure in the knowledge that it is much more likely that wrongs will be alleged against them under the contract than that they will need to allege wrongs against the contractee, and that businesses feel that arbitration is likely to favour them. If businesses didn't believe at least the latter, or even if there were just divided opinions, then arbitration clauses would not be included in every. Damn. Contract.
I agree that the kind where both parties come together and agree, for a specific case, to engage in arbitration and be bound by the result is OK. The kind where you agree beforehand, when no damages have occurred, that if any case comes up then you will both go to arbitration instead of court, is not OK.