Tons -- if you want to look at the evidence, look at the gender and ethnic makeup of programs for kids gifted in math in the US. First-generation kids are deeply overrepresented, and by the third generations kids of any ethnicity are all equally bad at math in the US. Cultural attitudes have a lot of influence on people's approach to mathematics.

Moreover, in my experience I see lots of people in the US say "I'm bad at math" if they are not good at algebra. Algebra is not math. Neither are fractions. Geometry is part of math. So are combinatorics, logic, numerical analysis, etc.