I have many physicist colleagues that are very successful in ML, and I think most of the people that work in the field today don’t have a formal ML education (as those just became popular/broadly available 5 years ago).

Deep learning brings some new paradigms to the table but it’s not fundamentally different from other optimization / ML methods, so with a background in statistics, math, natural sciences or quantitative economics (to name only a few) and a good understanding of programming (which most people from these fields have) you can do just fine. If you start out with zero knowledge of university-level math and no programming ability it will be much harder of course.