It follows by contradiction of the opposite statement: Lets say that all "you"s across all branches perceive all the other branches as well. That means that they do influence each other. In other words they are not separated and never where different outcomes to begin with.

But, all the interpretations of quantum mechanics start with the axiomatic assumption that the universe is modal and that there are different possibilities / outcomes. They only differ in if they chose to turn hypothetical outcomes into real outcomes or simply let everything be equally real from the get-go.

And yes, even superdeterminism has to deal with / model hypothetical outcomes. It just says that some of them cancel out each other early on as they would lead to inconsistencies in the future otherwise.

We are seeing all the other outcomes and branches too, we just aren't aware of it since those are separate universes.