Actually they're not immune to being thrown off, there was an experiment where people with perfect pitch were played a long orchestral piece that very slowly detuned, e.g. over the course of half an hour dropped one semitone. The study found this also detuned their sense of absolute pitch, not just immediately after but also months later.
https://www.psychologicalscience.org/news/releases/perfect-p...
He was one of those little kids who had absolute pitch from the start, though, and he definitely views it as a two-edged sword!
I don't have perfect pitch, but I actually seem to have anchored on A=443Hz. When my violin teacher insisted on 440Hz it would cause me discomfort the same way you describe, just feeling wrong. And I've found that when I try to tune an instrument from memory, I'm consistently sharp. (Assuming I'm not off by a whole-tone, because I don't have perfect pitch.)
Maybe you could apply their methodology to yourself to retrain your A to 440Hz?
Vienna Phil have historically gone even higher, but are probably down to standard levels by now.
I knew many more people with very good relative pitch. In school those people would get a middle C at the start of ear training class and ace every note after that because they knew where that C was.
Not least because the argument about "what note is this" and "what note do you think you're singing" starts basically day one of singing lessons. Trying to learn any practical music when you can hear the difference between close frequencies mean you're effectively learning colours from a group who insist that blue and green are the same colour.
One of my party tricks used to be saying the song and artist from the first 5 seconds of a track. These days with samplers and synthesisers that's almost impossible. I still get tripped when I hear some samples, like "that's the bass guitar from {song x} WTF" in the middle of some otherwise pleasant song.
I didn't recognize the notes like I do on bassoon. I just knew. I sometimes still do, but not every note I hear. Now it is just once in a while.
I had an ear training teacher who would play a bunch of random atonal notes on the piano between exercises to "reset" our ears. Only works for relative ears.
Similarly I (relative pitch) retain the key of a piece I practice. But if someone plays a random sequence of pitches it pretty easily makes me lose my anchor or at least make it lose focus by a semitone or three.
Whereas the "real" perfect pitch people are always anchored, no matter how much noise anyone throws at them. Which is also a source of discomfort and difficulty, for example when they need to transpose or work with a different A4 than their "internal" learned one. Where us relative normies would just shrug and accept the new A4.