Sex is purely genitalia. The two extremes are male and female, but this is a spectrum and not a binary.
Gender is all the other shit that society piles on top of that. The two extremes in western culture are masculine and feminine, but this is a spectrum and not a binary. That this mostly lines up with the sexual spectrum is an arbitrary decision by society; society could define gender in such a way that it is orthogonal to sex. Examples of where gender doesn't line up with sex in western society include butch within the lgbtq community: http://en.wikipedia.org/wiki/Butch_and_femme. Other societies have in fact defined gender in a way that doesn't line up with sex, so this isn't just theoretical: http://en.wikipedia.org/wiki/Gender_identity#Non-Western_gen....
In the thought experiment he is not being rejected because of his sex. He is being rejected because of the gender he presents is not sufficiently masculine enough.
This has everything to do with his sex. He is being rejected (hypothetically - within our thought experiment here) because he is considered to be an insufficient exemplar of the male sex. To phrase it a different way - he is insufficiently masculine for this cheerleader (i.e. a very high-status female) to have an interest in him.
You correctly point this out later on - She's rejecting him because he doesn't have enough of the traits that society says a man should have (and maybe has too many of the traits that society says women should have): He's not strong, he's not assertive, he's not domineering, he's not good at arbitrary athletic competitions, etc. etc. etc.