The practical difference in the listening experience is, as you infer, negligible, to almost everyone. Classical music, or other such audio with large dynamic ranges, suffer the most. And while those frequencies above 12-16kHz are often the most obviously hit, the entire range suffers in other ways, phase, low end smearing, transient loss. And while first generation lossless to lossy conversion really is an excellent trade off, lossy to lossy very quickly produces audibly worse sound, even 320 to 320 conversion. But you knew that.