Reflecting on this stuff is my golf...
Even if I agree that THE number "one" is "just another broad category that a single apple fits in to", I think the epistemological status of the Broad Category is then itself just as problematical, existence-wise, as the original concept of "number", eh? We can at least display a pair of apples and "count" to "two" with them, etc. What hope do we have of enumerating (no pun intended) the Broad Category of "one"? And what do we gain from introducing it?
If you buy the above, you’d have to say that a single apple is also “one”. After all, it’s just another broad category that a single apple fits in to. If not, I think you’d have to reject the fact that a honeycrisp is an apple, which seems untenable.