The first problem is that the notation is usually "the first symbol that popped into some random genius's head 200 years ago". And once the notation is set, it's set, no matter how poor it is, or how many other places it's already in use etc etc. Then, as you note, sometimes there are multiple notations. Ugh.
The second problem is closely related to the first. Mathematical notation is write-optimised. This makes sense because of the long history. But that doesn't change that write-optimised languages are harder to read and understand, even for experts, than read-optimised languages.
In a programming context if you today reduce all your variable names to single latin letters and all your function names to single greek letters, you will be widely mocked and reviled. In maths it's Just How Things Are Done.
I guess what I'm saying is: the curse of mathematical notation is pen and paper. The boundaries of QWERTY liberated (almost all) programming languages from the curse.
I actually really like mathematical notation. Once you get used to it, the terseness can make things a lot clearer than natural language.
(If you've ever tried to read old mathematical articles / books that are a few centuries old, you'll understand the power of mathematical notation. Check out "God Created The Integers" or "On the Shoulders Of Giants" by Stephen Hawking if you're curious: these are two books that provide excerpts of highly influential works from earlier mathematicians and physicists, respectively)
Serious question. What do you think is the general impression of APL programmers? =)
I already compromised by adding "today" and "almost all" as qualifiers.
The problem isn't the notation per se, it's that teachers don't spend nearly enough time explaining the notation itself. It's a foreign language that they are so skilled at that they don't understand how unfamiliar it is to their students.
I was well into a physics major before I really stopped and carefully considered the many different notations used to represent derivatives (dx/dy, f'(x), y', y-dot, Dsub-yX, etc.) I realized that I had developed separate context-specific bodies of calculus knowledge/skill from different fields with different notations and approaches, and that these were all the same thing. Different techniques in different contexts were an artifact of different styles of notation, not differences in the actual math.
I can understand Italian to some extent as a side-effect of my study of Spanish. If I took an electronics class in Italian, I would understand some of the concepts and misunderstand others. Would my troubles to understand certain electronics concepts be due to trouble with electronics or trouble with Italian? Who knows? Both types of misunderstanding would compound each other.
If teachers spent more time carefully teaching this foreign math language before (and while, and after) using it to teach math, I'd guess a lot of students' "math problems" would magically disappear.