There's no direct observation of addition, it's just a symbolic construct to model something we observe. I suppose a reality where such a model never made sense is possible, but it would be a very foreign environment that is completely outside the scope of our intuitive understanding. If it made sense only for a subset of the cases it does in our current understanding, then that would be the context in which it was discovered/understood.

Addition is defined in such a way that the universe it's performed in doesn't matter (in fact, no universe is required at all), which is true for most if not all of formal mathematics. See set theory 101 for the very beginning on how's that possible.