You give rights to people who offer money, not to people who offer goods and services. Why?

Also: what happens when both parties are offering money (e.g. currency exchange)? Can both of them choose who to transact with?

And if neither party has money (e.g. barter) then I assume that you think they should be coerced into working with each other?

No, you can always choose not to interact at all. You can always choose not to offer a service or product, or not to use a service or product. I just think that the larger the scale at which you decide what to do, the less interests that are not directly related should be permitted to matter.