You're not alone, the narrator is a very critical experience of listening to audiobooks. If the narrator's voice, tone, or style aren't appealing, it'll ruin the entire experience. The problem is that this is largely subjective - there won't ever be a narrator that'll make everyone happy but like you said, most are good enough with the occasional bad apple.