If I remember correctly, some of it came down to encoder settings as well. Joint stereo, for example, always sounded janky at all bitrates. CBR vs VBR also made a difference. Also, the quality of the source was a factor. A low quality source that was already kind of lossy would sound much worse than the source itself when encoded at lower bitrates.