There of course needs to be a common set of ideas to communicate, but I'm not convinced you need one particular set. In many cases, having some of the ideas means you can synthesize the rest. You could for example reason about numbers by drawing upon size rather than quantity. Then you basically end up with the Euclidean method, a mathematics of proportion that goes a surprisingly long way.
If there is any primal, axiomatic concept, it is "object"/"thing". From object you can derive quantity (many objects), from quantity you can derive quality (two objects are different), and so on and so forth.
In some languages the name for door is the same as the name for mouth, and a door is a bit like a mouth so it makes sense to relate them. All language is metaphorical like that, except not always as explicitly. When we say a thing is an object, we say that our concept of that thing shares similarities to the concept we have for objects.
Yeah, nothing is stopping you from building towers of abstractions that are N degrees from any real experience or data. But those heavily-derived concepts are prohibitively difficult to teach and communicate, because each level in the abstraction hierarchy adds semantic noise. A major reason why classical, "pure" mathematics pedagogy is infamously ineffective.
Ultimately, every great mathematician learned to count with rocks or apples -- not by internalizing Peano's axioms.