More importantly, the problem isn't trying to hear 96dB of range all at once. The baseline is fixed in place, so when you have a quiet section all those high bits are 0 and you need the low bits to have enough detail by themselves. And a whisper is pretty far from the quietest thing you might have in a track.

If you can hear the dithering noise, then I'm pretty sure there are sounds you're hearing that would be wrong or missing without the dither.