Whoa! I've been so annoyed by this for years, so interesting that you figured it out. It's the kind of inelegance in design that would have had Steve Jobs yelling at everyone to fix, just ruins immersion in music and had no obvious way to fix.