Yep. I remember the early visualizers did indeed display something akin to frequency spectral data. I wrote a few visualizers for SoundJam (later iTunes) and the API was exactly that: left and right bucketed spectral data.

I know, bar graphs got a little tiresome so people tried writing more interesting stuff but at some point the "visualizer" became so disconnected from the music that even if you paused the audio the fireworks and "flow fazing" just kept right on doing its thing.