A First Course in Probability: http://www.amazon.com/First-Course-Probability-9th/dp/032179...
Linear Algebra with Applications:
http://www.amazon.com/Linear-Algebra-Applications-Otto-Brets...
Note that the Bretscher book has really terrible reviews, but I can't evaluate if the reviews are correct or not.
Linear Algebra with Applications: http://www.ebay.com/itm/Linear-Algebra-with-Applications-5ed...
International editions of textbooks are immensely cheaper.
Many of these things are described in the comments. I almost exclusively buy international textbooks for home reference if available. The price difference and the relatively small quality difference makes it a no-brainer. If you are doing it for a class though, find a friend with the overpriced version for homework.
I took a course on Linear Algebra (Bretscher's book up to chap 9) and a Probability course (Ross' book up to chap 6) and did very many problems by hand on paper. I just finished a ML course (Bishop's book, and Jordan's book), mixed grad+undergrad, which was 80% problems w/ pen and paper and 20% code up something algorithmically trivial but mathematically challenging, and don't think I would've been able to pick up the additional math along the way without these two great books and their many exercise problems behind me.
I read layman's explanations of ML concepts a year ago, and got nowhere in terms of my own implementation+debugging/improvement upon established techniques. Now I can solve problems I saw a year ago and thought "no one can do this."
My advice is to take the gateway drugs first, Probability (Ross <- I love this book!) and Lin Alg (I like Bretscher much better than Strang, but not everyone agrees with me :) Take a course in real life (for a grade and a transcript) at a competitive university if possible, nothing makes you study thoroughly like a gun to your head.