Maybe by "number" in general do you mean something that encapsulates both "real numbers" as in lengths, and "integers" as in the discrete counting numbers? In which you can quite easily do so by defining real numbers, either formally or saying that something like the Wikipedia definition that they are a distance along a line, and then saying integers are a subset of those numbers.
I agree in a sense there is no agreed-upon simple non-formal definition which encapsulates all of those together but I think that's unfair to try to do since hyperreals, quaternions, etc are extreme extensions which aren't used in most ordinary mathematics.
That is like saying that physicists have no precise definition of "distance" because there is Euclidean distance, Geodesic distance, Hyperbolic distance, Hamming distance, Levenshtein distance etc etc etc
You can go down the same rabbit hole I guess and say "yeah! true! distance has no definition either!" but I don't find that very helpful. At this point you are just saying nothing can be defined, at which point the phrase "we have no precise definition of X" has no meaning as it is true for all X.
Similarly, "crazy" is not a term of art for psychiatry. That doesn't mean a lack of definition, and lacking formalism does not mean inadequate for communication. By using a fuzzy term for communication, you imply a taste in curation and abstraction. It means you don't want to get into the details of what crazy means.
> That is like saying that physicists have no precise definition of "distance" because there is Euclidean distance, Geodesic distance, Hyperbolic distance, Hamming distance, Levenshtein distance etc etc etc
A metric is a term of art, and if that's the level of specificity you wanted, then it's adequate.