This. Or to put it another way: "You know who's a more interesting person in adulthood? Someone whose parents didn't bow to peer pressure when they were a kid."
FFS. If you're a parent, you're supposed to be setting an example. Not giving in to the same impulses we ostensibly teach our children to resist.
You absolutely can opt out of socialization for both you and your kid. But, that is what you are doing there.
There's also, you know, letting your kid hang out with other kids organically. They do that naturally, you know?
If other parents expect planned meetups, then "organically" is exactly that. Even if they take no issue random meets in principle, they already planned most time for other things. Their kids are not outside to be randomly met. Often other kids don't even live within walkable distance.
For that matter 3-6 years old don't "organically" start to visit each other alone without parents. How exactly you expect that to happen, really?
The problem is that this behavior is driven by a tiny monitory, and supported by the rest.
The honest answer is to be the jerk who doesn't go along with it. The only way we can stop being victim to societal trends is to go against the flow.