Again, this really depends on how mathematically competent you already are. I'm just basing this on how I felt coming to the course after having finished my degree about 10 years ago, therefore not really having most prob/statistics fresh in my mind.
It's the very first bit of the course, I think everyone who is interested should try learning it. If not it's fine, but I wouldn't want anyone to not even try to spend a few hours on it because someone on the internet said it would be too hard.
My worry is that people will be put off from the field of machine learning if, 3 lessons into Andrew Ng's course, they will see that they don't understand anything, and that it's not practical to boot.
So my advice (generally applicable) is to try a few different things, because different resources click for different people.
If you never took calculus it's probably going to be hard going, but almost all modern machine learning requires basic calculus.
I would really recommend going through the first part of the course about linear regression if you haven't encountered it before, it was really eye opening for me.