I'm french, so I may miss in what way the language is difficult.
I agree, there is a lot to learn to speak a language correctly. I was very average at english at school, but I improved by reading and recently listening everyday. I'm not going to write novels, I'm interested at communicating with others and as long as I can do that, I'm fine.
It is just a cognitive stuff - it is very difficult to distinguish where the one vowel ends and next one starts when you're listening to a native speaker and almost impossible to articulate correctly such difficult and beautifully sounded language if you started to learn after 25-30 years old.