Compare, say, Brad Pitt to Bill Gates. Or The Rock to Peewee Herman. There are differences between how closely those people fit the archetypal mold of "manly".

I'm comfortable saying that Arnold Schwarzenegger is manlier than I am. He ticks more of the boxes we associate with "man", culturally. He's muscular, successful, attractive, tough, etc. I've got a bit of a spare tire, I'm pretty soft, etc.

He and I both are male, and about the same degree of male. We both have penises, etc.

In this way we can use "man" to describe our features that are culturally associated (ruggedness, toughness, cigars and whiskey, etc.) from the biological features (penis, testicles, body hair, etc.)