Sure, in the moment, I'm not playing against my opponents best choices. But without take backs my opponent is developing the skill of thinking before acting, which makes him a stronger opponent overall.
We only allow take backs for:
1. Newbies. They don't know the rules, so we allow take backs during their learning phase, and also help them understand why the thing they did was non-optimal. This lets us play the game while also helping newcomers learn the rules and general strategies quickly.
2. Misplay. If another player does something not allowed in the game (e.g. plays out of turn) which changes a second players understanding of the current state of the game, we let the second player take back their move if they made it before we discovered the flub.
But chess also has its version no take back a in touch-move requirements. Part of your"best" self is making good decisions the first time under pressure. Or they I'm not playing you, but a small group of you (often with other played input).
Competition involves the stress of making decisions that you cannot take back.
If you reject take backs they you aren't playing the most competitive version of your opponents.