I've spent quite a bit of time in New Zealand, and a common complaint I've heard is that when they travel to the U.S., people often ask if they are from Australia. When I first rocked up to NZ I couldn't tell the difference either, but now the two accents are like night and day. Beyond that, every country has its own regional accents as well.
And to be fair, I reckon a good number of Americans who haven't traveled can hardly differentiate accents beyond American English vs non-American English too.
We have somebody in the family that is know as:
"Blah ahk guak fiuya sdsds caramañola ñame!". And " caramañola ñame" is the only thing you will understand 100% of him ;) -- the other text is not a perfect rendition. In fact, nobody agree how exactly him talk when left him at full speed!
How fast is the talking is not the only issue. Is the slang. We can have the same language... but, o boy! the slang is diverse even in the same country.
And some words have reverse meanings, or don't make much sense.
For example:
"Esta haciendo sereno"
"Is doing serene (???)"
If use this dictionary, you could get it wrong:
http://www.spanishcentral.com/translate/sereno
Instead, despite the fact "sereno" is exactly "serene", this in colombia mean "colder wheater".
Is so funny when we teach the slang to outsider. Is crazy fun for them ;)
No, it just means that you need more exposure. That very same thing happened to me while learning english (as my 3rd language after Spanish and Catalan [0]). I started to interact and shared flat with people from USA, England, Wales, Australia, Ireland, etc... For instance I wasn't able to differentiate between Scottish and Australian, only between American English and non-American English. Now I'm quite good at it.
[0] https://en.wikipedia.org/wiki/Catalan_language