"Which of these parties don't you consider being a part of society?" - This is a strawman. Both are part of society, but the contract between them may do more harm than good in total. And blindly holding contracts sacred quickly leads to bad outcomes (non-competes, EULAs, some NDAs, etc.)
Man, the straw man fallacy on HN is like the proverbial hammer... You are talking about externalities of private contracts. Thing is, this also is subject to specific legal regulations in most Western countries. It is not subject to whichever moral judgement you are trying to fit in here.
Construction, industrial emissions, things like that are when the state can limit the civil rights of contract parties. Buying and selling bonds is generally considered not to have externalities that endanger the common good.
Also, contracts are between at least two parties. Which of these parties don't you consider being a part of society?