I think the answer is that "looking pleasant" is much more of a goal for commercial software. In usability terms the much denser GUIs of the 90s were arguable better in some ways than what we frequently see today, especially for power users.

Yes just like the design of car infotainment systems -- looking good resulted in the touch screen era. Luckily we're finally realizing our mistake and a few automakers are putting old school knobs back in.