To use your analogy, let's say Bob is constantly stealing from you. You urgently need some essentials, and you say, "I know, I can steal them from Bob." But somebody stops you and says, "Wait, you don't want to do that! If you steal from Bob, he'll steal from you!" You would probably not find this persuasive, because you know Bob will steal from you regardless. There probably are good arguments you can make against stealing from Bob, but that particular one doesn't jibe with reality.
You're framing this as a simple prisoner's dilemma problem with sharp-edged binary choices - but it's much more fluid than that in real life.
The fluidity non-binaryness of this is even more apparent given it's millions of people we're talking about, over time lasting generations.
If life really was a prisoner dilemma cooperation would be impossible.
But this isn't really how it works. Forebearance actually does have value because it changes society as a whole, shifts expectations at a broad scale, and creates moral authority to demand better from others which is persuasive and powerful.
At the end of the day, what kind of society do you want to live in? Will you contribute to making that society, or not?