There might be also factor of women not getting benefit from associating with men.

You can't assume that all decisions were made by males only.

I think it's both; I spent some time in Kyoto and have clearer perspective. Both genders there grew apart from each other, pursuing different, incompatible goals, with huge cumulative effects on society.

I've seen a few too many women where they honestly are better off when they don't have a male romantic partner: the men they choose end up being a burden to them. For younger women, it's because they keep choosing men who just cause them problems and can't support themselves. For older women, there's a common pattern where the man dies very quickly when his wife dies, but if the man dies first, the wife can live many years or even decades because now she's free, and doesn't have to take care of him.

Honestly, the whole 2-married-partner relationship thing seems to only work out well in a small minority of cases.

Honestly, I can't blame a lot of these men for feeling that way and not wanting to feel like they're repressing themselves. In some ways Japan's gender roles can be quite conservative, and I wouldn't blame someone for saying it's not for them.