As someone who has seen enough of the math, I strongly feel that probability just makes ideas seem "sexier" in ML; often you don't really need probability, or you only need it to use Jensen. Worse, using probabilities generally masks deeper ideas from surfacing. Bayesian-ism, has become another cargo-cult, in many ways.