If the two people involved in the two people involved in the trade are exactly x better off and the, lets say, two people not involved in the trade are exactly x worse off, that transaction is still zero-sum.

Zero-sum games are not limited to two player games.