I'd say say the truth is somewhere in between. There are many people who think math is hard, but they can be taught quite easily with a skillful teacher if they're motivated. However, there is a group of people who simply can't grasp it. No matter how you explain, they stare blankly at you and say they don't understand. It looks like they know what you mean if you use a simple example "You have 3 apples but want to have 13 apples; how many are you missing", but make it just a bit harder and you get back to square one. It's just a kind of people. They're otherwise nice to get along with, they seem to understand many things about life in general etc., it's just this aspect that is somewhat blocked. I hope one day we discover a safe way to somewhat activate the neurons in the related area of the brain to break this barrier.