It's more from ignorance than anything. I speak Quebec French and had little trouble being understood in France in the two times I've visited. I have folks from France, French Africa, and Haïti here at my job. It's more just ignorance about the accent than a lack of actual experience not being able to understand it. It's certainly not analagous Californian English to Shetlandish Scots, far from it.