That's an interesting one, "imagine if we could all agree not to violate each other's rights". Thing is, rights aren't fixed and absolute, unless stated so vaguely as to be useless, like "the right not to be coerced". It wouldn't just be a problem of trust and consensus, but of shared knowledge. I mean we'd have to have consensus about what our rights are, and more specifically, what constitutes a violation. Not to mention consensus about how reality is. So you get all these edge cases where clarification is needed, and that's why we have law courts: legal cases that set precedent are edge cases. Also, the situation, I mean culture, keeps changing, and we have to invent new rights to keep up. We're going to continue violating one another's rights so long as our mutual understanding is imperfect and so long as we're not a hive mind.

Not sure if this means we have to have locks, prisons, and militaries for all eternity.