I think it's a decent point. Maybe we generalise from "the sort of people who read books" suggesting it's something virtuous. Or we have certain books in mind (rather than trashy romance novels).

Reminds me a bit of the idea of tea drinkers as some cultured ideal.