Sounds like you could adapt Leibo’s sequential social dilemma games. https://arxiv.org/abs/1702.03037

Instead of a total collective reward being the goal, you’d have team-based scores. You’d need cooperation within the team, and aggressive action against the enemy team.

Here’s an implementation of 2 games https://github.com/eugenevinitsky/sequential_social_dilemma_...