i dont think that's true. A lot of people would assume that this "mathematical game" is like theories with air pressure. The statistical methods used for computing heat and pressure can be "explained" as little particles of atoms moving about due to their vibrations, and the ultimate observed behaviour is just that - a lot of particles pushing and shoving dependent on the temperature of these particles. this simple explanation can be understood without the maths - after all, the maths is only there to tackle the large numbers and statistical inferences needed. The statistical methods "isn't the theory".
However, such views restricts the mind imho. To seek a simple, understandable explanation that is not mathematical is probably a wrong goal. A lot of people have tried to explain laws such as gravity using such simple rules - a particle that moves and hits in all directions, but bigger bodies will block certain direction (thus resulting in a net force from the opposite side - aka, gravity).
The idea of their being both a wave and particle based description of photons that don't mix well is basically physics as known in the 1920s. The modern theoretical framework of light is quantum field theory (special relativity + quantum mechanics, but actually not quite that for [insert qft textbook] reasons...), which doesn't fit into either of those old ideals particularly well because they are basically shadows of an underlying model projected at different angles.
Edit (further detail): The way physics is taught in British secondary schools is a tragedy - ignoring that you don't even learn the fundamentals properly (no calculus!) - you aren't given a non-mathematical outline of these modern theories. In their eyes quantum mechanics has not really advanced beyond the photoelectric effect, which is where unhelpful weirdness like you'll see in this thread comes from.