Because unless you're listening in an anechoic room - which is Not Fun - reflected ambience creates a hemispherical sound field around an acoustic source, and stereo recording only captures a small sample of that. (Enough to create a 2D spatial illusion, but no more than that.)

And also because loudspeaker technology is still pretty terrible compared to the rest of the audio chain. Converters are nearly perfect, amps can be made close to perfect at non-trivial cost, microphones are okay but usually add some colouration, but even the best studio-grade speakers have limitations and add significant distortion.

And consumer-grade speakers - even expensive ones - are nowhere close to perfection. And even if they were, you'd still have to deal with room resonances in the performance room, which would add extra colour to the sound that wasn't present in the original recorded ambience.

Basically hifi is good enough to be musically enjoyable in a near-enough not-too-distracting way, but you need research-grade technology - like complex wavefield speaker arrays and acoustic tuning of the performance space - before you can record and reproduce sound almost transparently.