Not being particularly interested and not being taught math is not the same. You probably had normal math just like anybody else and was not behind other people all that much.

But I agree that the tendency of programmers and public to assume that if you already don't know a lot about computers, you are lost case and stand no chance is harmful. There is strong tendency to overstate how difficult things are and that makes people look elsewhere.