You can't tell the difference on the vast majority of music. But encoders occasionally mess up, producing an audible difference. In the case of MP3 this often takes the form of a pre-echo. For example encoding cymbals and hapsichords seems to be difficult, though much less of a problem than it used to be. My understanding is that such killer samples exist for good MP3 encoders, even at 320kbit/s, but it has been a couple of years since I last looked into the topic.

(Personally I downloaded some of those killer samples and couldn't tell the difference, but other people reliably tell them apart in an ABX test.)