We don’t really. We have three receptors that respond to different wavelengths (plus black and white) and our brains stitch together a composite image.
Animals can have fewer or more receptors, and see more or fewer colours. Just like a person who is red-green colourblind, and sees them as the ‘same’, some animals and even rare people can see different colours where we see only one. We are all effectively colourblind to some extent.
What I personally experienced is there are some individuals who can identify specific notes down to the unit frequency (I played a 439Hz tone, the person said "Uh you're a hertz short" and I fixed the bug in my program). That level of ability is generally believed to be not learnable after the brain loses a certain amount of plasticicity.
Continuing from my own experience, people who do not have absolute pitch at that level can improve their skills in pitch detection including: identifying intervals, identifying octave, identifying common notes in an octave, and people like me who can barely tell you the interval between two notes can improve their pitch detection somewhat.
Whta is rarely or never observed it people with relative pitch gaining perfect absolute pitch after growing up, regardless of the amount of training.
And a musician probably couldn't tell the difference between = vs == or & vs &&. If you don't recognize that someone that spends all of their time doing something will be better at that something compared to someone else, then there's just a large disconnect. Also, the concept of "practice" yielding improvement is not a new concept, and I'm surprised it seems to be so shocking of a concept.
It seems you really wish to believe anything can be mastered at any level, but it's just not true - no matter how disturbing that might sound. In pretty much any kind of mastery you can improve to some point and then your learning curve starts getting into the saturation due to your various genetical and psychological limitations. Your gains start getting smaller and smaller, and since life is limited it also limits what can be accomplished during it. People who were born with certain talent/predisposition will always win in that race (presuming they also work as hard as you), simply as they start off from a better starting position. You can't just decide as an adult to become a new Usain Bolt or Jordan or Novak Djokovic. If you haven't already started training hard as a kid it's just too late for you, no matter how much you wish it. And perfect pitch is just another extreme example of that as it seems to be closely related to the phase of speech development in kids, which ends when we're 8-11 years old. Try to pick up some foreign language that you know nothing of, just by listening and watching people use it, without any other help. And then compare your progress to a 2 or 3 years old kid who does the same seemingly effortlessly, even with the most complex languages in the world. Rick Beato has an interesting theory that kids can be directed to develop the perfect pitch by exposing them to a lot of advanced music with complex harmonies and scales, as the brain - he believes - treats music same as speech and recognizes the pitch of a sound as encoded information. But again it works only with kids. From what I've read about it there's no a single case known that someone beyond doubt proved to have had trained themselves a perfect pitch hearing as an adult.
So I'm not arguing someone can train themselves to be perfect pitch, but in all practical purposes it's close enough. If you can pin point the specific player in a group that is off pitch, it is still impressive. Does the person that can detect the 1Hz difference bring any more benefit than the person that says 439Hz and 440Hz is the same note? If it does and you're depending on a human for that level of accuracy, I'd suggest you're barking up the wrong tree and should be using intstrumentation for that.
The link below is a study which shows that the distribution of pitch recognition among the general populace is bimodal (you have to scroll down a bit). This matches with my experience that, irrespective of practice, people either have it or they don't.
This is an issue in the interminable debate over Frank Jackson's "Mary the color scientist" thought experiment, where anti-materialists seem to think that if you cannot learn what it is like to learn what seeing colors is like from a science book, then materialism must be false. Presumably they would hold the same position over learning perfect pitch.
My experience with one kind of "magical skill" that software engineers have is: someone reports strange problems with some application (that you did not write). You watch them reproduce the buggy behavior and you see one step which seems "off" - not what you would have done. You try the same process but with the step you think seems right and it works. Person goes off happy that their problem is "fixed".
Now of course this is "learned" but in a whole-systems way that just looks like magic for someone from the outside. It's not an exact parallel, but I think it's an interesting one.
(Sorry I don't have a concrete example, but it happens with some regularity. Like "I think you should let the cable modem power up before turning on the other devices" or "That screen seems to be flickering a lot, have you tried swapping the power cord to the other side." or "I don't think you should be crossing those cables, try running them all parallel." All little by themselves, magic together.)
Except, I think every dev on here knows from "learning" the diff between =/== yet has had the typo error in an if test where == was meant, but ended with a single =. Yes, it gets much easier to know why things are misbehaving after experiencing it enough, but it did require that experience to really "learn" it.
Same with any skilled trade. You can learn it by watching or reading, but the real learning comes from the doing repetitively. Some might call this practice. Pilots call it hours on stick. Devs with enough of this are called senior. Of course there are people that are naturally gifted with skills that will excel more than highly practiced people, but that doesn't mean practiced people can't get to the same levels.
What makes a pilot or dev 'senior' is mostly a combination of a sense of what is normal (and which deviations are significant), and good judgement. While these are skills learned by experience, they still can, to a degree, be communicated in language, but skills like perfect pitch can only be described in language - if you get it wrong, no-one can explain what you could have done to get a better outcome.
This seems like hair splitting to me. You even refute yourself by including the word _rarely_.
For whatever it's worth, I heard about this like a decade ago, and nothing more has come from it, so perhaps you are correct in that the connection is speculative (the study only had 24 participants and more research needs to be conducted with large samples), but it's an interesting study none the less.
This is just argument by assertion.
Try learning to paint and you'll perhaps see that your perception of colour isn't as good as you think. I did and it certainly opened my eyes, pun intended.
As for recognizing pitches, it's a trainable skill. I learned to play guitar a while back and it was interesting to watch the skill unfold.
Some of the open chords started to appear to me almost as distinct as different people's voices.
The first time it happened I was listening to Paul McCartney's "band on the Run" and just knowing that he was playing C then FMaj7 (Well, the rain exploded with a mighty crash...)
I'm just ok at learning parts by ear, some people are on another level.
I think the most interesting thing that this points to is that there is probably a whole world of skills that one cannot even imagine until one begins to acquire them.
I remember wondering how my guitar teacher could transcribe songs so easily, but now that I am a 'stream enterer' for that skill, I can sort of see what that must be like.