For me, swearing (and hearing it) has a strong emotional component. While I don't consider it offensive really, there's a moment of distracting shock. Someone using rough language casually in speech is a little bit grating. Things tend to offend me based on their meaning, not on whether they're in my culture's fairly arbitrary list of "crude language".

In contrast, outside of English, swearing doesn't bother me much. No shock. No offense. Maybe that's your experience, as well.