That's just quibbling over what "your individual rights" are; where does the line get drawn between "exercising my right" and "having my rights infringed on by the actions of another." There is no shortage of harm done by "anyone in the market" today, whether it's currently illegal and we call it "crime" instead of just a person exercising their freedom, or whether it's harm that isn't regulated today.