> ... other people in the library are probably strangers

This is the key thing--our brains process familiar and unknown voices differently. Recognition of a familiar entity and the ensuing reactions are automatic, and they all have a processing cost.