I think it's fair to say an environment that is so quiet you can hear a pin drop (i.e. ideal movie theater quiet) is not going to be welcome by most, for the reason you cited. But the same effect can occur in a noisy place as well, if a single individual is sporadically more noisy than the background noise.

So it's more about achieving some constant noise level, above which no sporadic noises rise. IMHO, the "library/whisper quiet" rule achieves this.