I think you basically identified the root cause of the issue. We are, unfortunately, only slightly evolved chimpanzees; We would gladly form groups to fight another group when bananas grow sparse.

Had we evolved from bonobos, things might have been different.