Exactly, the best part about Joe is he gets out of the way and lets his guests do most of the talking and just gently moves the conversation along.
Mostly this, though he does call people out on things that he thinks are wrong. The Adam Conover episode comes to mind. It's a good balance though.
This is kind of how I feel about Tim Ferris as well. I think Ferris is pretty blatantly self-promoting during his interviews ("When I was writing 'The Four-Hour Body'..."), which turns me off. And he has a thing for Tony Robbins that is baffling. But he gets some fascinating guests, and he is very, very good at finding good questions to ask.
Most of the discussions on his podcasts sound healthy and sane, but sometimes he gets more wild/controversial ones which is always entertaining.
He just doesn't go to the extremes. No clickbait, no outrage, no suspicious promotions, he just feels down to earth which is refreshing in this day and age on the internet. It doesn't mean that he's always right, but he realizes this and doesn't try to push his opinions too hard.
I find that the "haters" that always criticize him are missing the point of the postcast which is to hear the guests, not Joe.