Keep in mind that 'right to roam' applies to unimproved and (some) agricultural land, and roamers can't disturb the right of enjoyment of the owner. This isn't your normal urban backyard.

Its more like someone walked through the parking spot that you paid to park in on their way to somewhere else.