I don't think an appeal to popularity is a useful or interesting challenge, even on topics so closely related to popularity like this. It's perfectly valid to have an opinion that some piece of entertainment media is "bad" while acknowledging that it is popular. There are plenty of reasons a non-fiction content creator in particular can gain popularity without being good at entertaining. E.g. if he gets guests that are perceived by a large group of people as challenging a status quo that they hate, he may develop a religious-like following among that group, and that certainly is viable outside the context of entertainment value.

For the record, I am coming from the perspective of having listened to very few episodes, but enjoying what I heard.