Absolutely agree. A few podcasts I listen to that I would consider very high production quality I can easily do 2x and some of the more casual content ones I have pushed to 3x. However, some of the fringe, sounding like they are recorded on a phone become difficult to understand even at 1.5x.
I can enjoyably listen at 2x for some people, but others I cannot go beyond 1.5x and I've been listening to audio at 1.5x or 2x for a long time.
It's much less about their pace too. It's overall line quality and how clear they enunciate words.
For my own recordings I try hard to make them feel good at 2x (even tho I naturally speak fast and manually edit out pauses). About 10% of the people who take my developer courses listen at 1.5x or 2x and have enjoyed the clear audio experience.