Another way to state the same thing: very few people enter a contract contemplating that they will be the one breaching it, so they are willing to agree to a system for efficiently adjudicating conflicts. But circumstances change and make breach attractive for a particular party, and at that point they will want to protect their advantage, fairness to the other party be damned.
I think you're justified in having concerns about arbitration, but I don't think the line that you're drawing is the right one. Focusing on the nature of the relationship between the parties might be a more fruitful approach. I don't see any reason why two sophisticated entities, truly engaged in back-and-forth negotiations where each has leverage, should not be allowed to agree to binding arbitration between them for future disputes.
For broad definitions of 'people', I don't think this is true. I am utterly certain that businesses enter into these contracts secure in the knowledge that it is much more likely that wrongs will be alleged against them under the contract than that they will need to allege wrongs against the contractee, and that businesses feel that arbitration is likely to favour them. If businesses didn't believe at least the latter, or even if there were just divided opinions, then arbitration clauses would not be included in every. Damn. Contract.