I feel you...I'm not German myself. It's quite annoying, and every time I want to be sure that the number I speak is understood properly, I give it number by number. I've seen too many Germans having issues with that myself, and it's just too risky with foreigners.
Interestingly, I've rarely seen it being an issue with dates. Maybe because the amount is too small and people got used to it.
Many traditions have their quirks. But at least very few German-speakers will try to defend this format.