The thing is, we have so much brilliant art and literature available that you need to filter it somehow. The likelihood that something released this year is better than something that has been heralded and great for 100 years is very, very low. Not impossible, mind you. I've definitely read modern books that have had a huge impact on me (including, ironically, Taleb, who introduced me to the Lindy effect), and art and music have a certain cultural context that can make modern expressions more relevant.
I just sell my books quite frequently and keep only the ones that resonated with me. I'll also buy books after having read them as an ebook
So on the one hand recency bias results in us giving undue attention to new books. On the other hand we are too forgiving to classics. There is no lesson here. Finding the right books to read is hard and you can't trust reviews.
Yeah, I've struggled with this a lot -- e.g. still having trouble reading the works of Dostoevsky because of those endless walking-in-the-park-and-philosopising dialogues. These seem alien in my time and culture.
I suppose you have to sense the characteristics of that 50-year-old language -- and see the universal, timeless ideas behind that language. Language is a tool, a form. It does change quite a bit in 50 (or 100) years, whereas the general ideas change much less IMO. To a large degree, this is a "seeing forest for the trees" thing.
But yes, of course some modern books will end up standing the test of time as well. But it's hard to judge that a priori without actually reading the book yourself.
You will spend a finite amount of time reading in your life. If you read Dark Matter now, that could for example mean that you will never end up reading Dune, or The Count of Monte-Cristo. If you're gonna sit down and read, an older book is a much safer bet if you want to maximize the value of your limited time.
Still, modern books can sometimes offer other qualities that simply don't exist in older books. Sci-Fi is a great example of that, where scienctific developments affect the writing deeply. So (unfortunately) in certain cases you might have to gamble on a newer book. But I try to avoid that.
> an older book is a much safer bet if you want to maximize the value of your limited time
I 100% disagree with that. Value is highly subjective. I've read a lot of the books that are considered as "the most important books from the western hemisphere" and let me tell you, most of it is garbage. It's just a classic because back in the day it was novel and exciting, nothing else like it. You can have a good time with both, modern and old books. It's about your selection process, not the age of the book.
Reading has multiple purposes. Entertainment and education, most importantly. So I read books depending on the goal I want to achieve.
PS: if you read the commercial garbage from reading lists and what's new, you'll have a bad time, granted. As everybody can write a book nowadays, quality is harder to find tho.