Music notation does take time to learn how to read well at, but it's no different anything else that takes time to learn and master.

Once you get past a cursory "eff this" reaction, you start to see how downright brilliant notation is.

The vast majority of music focuses on 7 notes at a time. If you alter a key signature, you are playing 7 other (non-distinct) notes. Music notation encapsulates this concept very well.

That's only one example, but telling musicians their notation sucks and needs to be fixed because it's hard for a non-musician is akin to a musician telling a programmer that Python and Linux needs to be fixed because it doesn't look like a violin.