I think their argument may stem from the fact that they can’t remove themselves from their own frame of reference enough to understand yours. In their head, to have this abnormal diagnosis, you must either be exactly like their perspective of someone with aphantasia, or be someone who just doesn’t understand that your experience is just like theirs.

The problem with having a diagnosis like this is that a lot of people won’t understand what it’s like to not have the characteristics we take for granted to the point that it seems unreal. To avoid accepting that the scale of people who have experiences that they’ll never be able to understand is bigger than they imagine (and avoid cognitive dissonance), they must invalidate your experience.