so all thats necessary for someone to inflict pain is to claim they personally enjoy it?

No, they need to convince a large portion of society that it's good enough to allow. So far the arc of human history has shown this to be a difficult argument to make at scale and at length.

but "they" don't exist to the agent inflicting pain. "they" are just a number, by definition. Just the result of a calculation...

Edit: or if you dont like that explanation, why should a consensus of a calculation have any more merit than an individual calculation?

Well no, the agent's model of the other person is just data, but this model is so convincing that most people don't perceive it as such and they're thoroughly convinced of the realness of the other person.

Consensus matters because if people around you disagree with you they will kill you or put you in prison. At rock bottom this is how all laws and ethical norms are created.