If it wasn't a property of natural objects (in some way), then how could our predictions work so well in the real world?
It works for electrons. But when was the last time that you interacted knowingly with a single electron?
For example, all the predictions that make us capable of building skyscrapers that don't fall down for centuries. Does it matter that two bricks are not "the same piece of matter" if our predictions work the same for both of them? In terms of their behavior under particular circumstances, they are the same.
No, because of a dense, interconnected web of other social truths (the rest of the arithmetic model), the relative error of this one truth/model is negligible.
However, confusing your model for reality is a fallacy perhaps older than time.
> However, confusing your model for reality is a fallacy perhaps older than time.
I think that the human perception the world is a robust enough model to be equated with reality without issues. If you go down the path of denying perception, you might as well go full solipsism, in which case it doesn't even make sense to discuss reality at all.
The model is "good enough" for the purpose of creating a building, but that doesn't make the act of "creating a building" any less real, nor the underlying rules that govern matter any less true. Our descriptions are not real - the rules themselves (which we may not know exactly) are real.
Therefore, mathematics - the set of rules that corresponds to how reality works - itself exists in reality regardless of social consensus. Society can't change them by making a different consensus.
Bricks are in the category of "made objects" not natural objects, which generally means that they are _designed_ to come off a production line as similar as humanly possible to the other products. "My iPhone is physically interchangeable to yours" is a statement about the huge efforts of industrial manufacturing to standardise matter, not about the natural world.
Bricks too have quality thresholds that they have to meet or exceed. That alone should tell you that treating them like integers is a convenient abstraction, nothing more. The sibling comment has it right: counting bricks is a great model, but confusing your model for reality is still an old error.
That depends on if one of the rocks breaks in half as you throw it onto the rock-pile or not. And also if the resulting piece knocked off is large enough to pass your fuzzy and contextual distinction between "rock" and "pebble".
But IMHO, arithmetic such as counting numbers and 2 + 2 = 4 are not part of the natural world. If I have a rock and another rock, I can with minor effort tell them apart: they have different weight, size, shape, density, composition etc. They are each unique individual assemblages of huge numbers of atoms in distinct never-to-be-repeated arrangements. In what way are these 2 unique things "the same" ?
If I have an apple and you give me a frog, I have an apple and a frog. They're not the same. If I have apple A and you give me apple B, do I have 2 apples? I have unique apple A and unique apple B. We can pretend that they're the same if you like, but that category is in our thinking, not in the world, and we also know that we can also notice differences between them.
tl;dr the natural world is not fungible, but behaving as if it is, is a convenient abstraction for mathematics and commerce, not a property of the natural objects.