I can say that struggle is not universal: I learned German as an adult, and don't struggle with writing down numbers I hear spoken. In fact, I'd never even thought about it being hard.

Every once in a while I say a number backwards (like once or twice a year), but I usually catch myself half-way after spitting it out.