There's no direct observation of addition, it's just a symbolic construct to model something we observe. I suppose a reality where such a model never made sense is possible, but it would be a very foreign environment that is completely outside the scope of our intuitive understanding. If it made sense only for a subset of the cases it does in our current understanding, then that would be the context in which it was discovered/understood.