There are maths and proves behind Neural Networks, but Machine Learning is an umbrella term. There are areas within it, maybe not that mainstream, that are also based on solid maths foundations.

One of the most interesting and elegant examples , Topological Data Analysis that is based on Topology and utilises tools like persistent homology. They can be applied to image processing, classification, etc.