And there's more too it, of course. I chose this link primarily because I found another observation very interesting: "In many studies, [the] mother’s education [level] had a more significant effect on children’s scores than income."
Equating inborn aptitude to SAT scores alone seems very short-sighted.
However, stereotype threat is not the only mechanism that fits the poster's description. Most Americans, upon hearing that I'm a mathematician, say, "I hate math! and I suck at it." Then they stare at me hostilely or say, "You must be really smart." There's a little subset of 40-65 year old women, though, who say wistfully, "I loved math and it was my best subject... but... in high school they said I wouldn't need it so I couldn't take the calculus class, I had to take typing instead..." Tracking in high school is used extensively and dissuades people tracked into crappy high school STEM classes from pursuing STEM degrees in college (or getting in at all).