Also having grown-up on the coast, since I was a child I've always been in the water with all the other kids, without parents around, never had any life-threatening cramp, never seen it happen either. Maybe I've just been lucky and you've been unlucky, since it's all anecdata anyway - but it's strange to me how it can both be so jarring for you to do it alone, and jarring for me to be afraid of doing it alone. For me it's like you said it's dangerous to walk in the park by yourself because a tree branch can fall on your head. It's true but I will eyeroll almost unconsciously.
I have lived 15 minutes from the ocean my entire life and I have never gone swimming in it alone.
I'm not an absolutist either, each situation has to be assessed on its own merits. If the kids are all strong swimmers and they want to go to the pool without an adult, that's fine with me. I might even let them go together into the sea depending on which beach they're headed to. But if they're taking the 3-year-old in water wings out to the open ocean I'm going to be tagging along.
Likewise, if you want to walk alone in the park I think you'll probably be all right, but if you're hiking 10 miles out into the wilderness I'm going to strongly suggest you bring a buddy, whatever your age.
I was thinking about it more and I think one aspect that is important is what you mention about "which beach". I agree with you completely, there's beaches in our coast we'd never swim in, not even in a group, even as adults. And in some areas of the country almost all the beaches are in that category. So I think you're quite right, your position was more general and I was thinking too much about the beaches I had nearby while growing up. Particularly how many meters of "foot" (not sure what the word is in english) you have - ie can you just stand up and not have to stay afloat.
That was always the understood rule as kids, we can go alone to the sea but we need to stay "with foot".
If I was walking on the beach and I saw someone swimming alone I probably would hang out in the area and keep a passive eye on them until they got out. I wouldn't feel the need to get in the water unless I saw that they needed help.
You'd be surprised at how rarely this happens, though. Usually there are lots of people at the beach.