My spouse is similar - he finds women easier to be friends with. I'm the opposite - I tend to get along better with men than women. (We are an opposite-gendered couple, btw). To answer your last question: Yes, for some reason there is something controversial about men and women being just friends and I wish folks would stop worrying about it.

I got a lot of crap when I was younger because of my friend choices. Teenage years? My parents wouldn't trust my friends. Didn't like me hanging out with them. Folks assumed I was having sex with this group of guys, which was really weird as I didn't generally date within our core group. In many areas, it simply isn't socially acceptable. Yet it is completely socially acceptable to be somewhat possessive of your romantic interest, up to and including forbidding them to have opposite-gendered friends. For men stuck in a "macho" society, it can signal that they might be gay and made fun of for it (and I don't actually understand why). It sucks, and I'm quite happy my spouse isn't one of those people.