That's a very strong statement. There's domains where machine learning tends to have better predictive performance than (Bayesian) statistics, but the converse is true in many other domains.

I would summarize it as Bayesian methods work best in areas where there's often not enough data, there exists significant expert knowledge, and you can properly specify a model. And yes, they do quantify uncertainty.