Funny thing about quantum mechanics... we use numbers to describe probabilities and functions to describe probability distributions. But the mathematical models we use are incomplete (and, looking at neutron decay, incompatible) so can we really conclude that numbers are the right abstraction?

And no, not poems. Elements of finite groups are not numbers, but they crop up in physics frequently. Topologies are not numbers, but they're also significant in physics.