There's a real difference in that for a speaker of a language where those are two allophones of the same phoneme, butter and bu'er mean the same thing; they're the same word pronounced differently. For a speaker of a language where the difference is phonemic, they're different words pronounced similarly.

That matters if a speaker of the former language tries to learn the latter. It's hard to learn to reliably make a phonetic distinction that has never mattered for comprehension before. See also Japanese speakers struggling to distinguish "rock" and "lock" in English.