The difference is that because Youtube has no moderation in its comments, smart people look at the comments and immediately decide that it's not worth their time. So, the only people who comment are the idiots who just like to see their words on the Internet.
I'd argue the root problem of YouTube comments is that they're not associated with any real community at all, in most cases. There's almost no benefit to posting a good comment, and no penalty to posting a stupid one.
Imagine how stupid the comments would be if everyone watching a TV show with you could post to the same wall.
Most of my programmer friends listen to podcasts, but I honestly have no idea what smart people do with YouTube.
The thing is, I don't doubt there's lots of interesting content on YouTube. I just can't conceive of browsing through it for fun. There's too much good stuff to read.
Anything that requires actual reading already has a barrier to entry to the dumbest of the dumb.