But are they? Unless I'm misunderstanding what you mean here, I'd say the opposite is clearly the case - learning "A is B" doesn't automatically mean learning the reversal; you either have to learn it from external source, or infer and memorize it - both involve an extra cognitive effort. This maps to LLMs as training, or "in-context learning" and then adding the inference to the training set.
In the non-simplified case I'd actually say that we have some natural experiments, but that might spoil this as well because you can in fact say that this is training for such invariance (which I'm unsure is problematic to my claim but pushback is definitely welcome). If we're learning history we'd often learn information such as "On December 7th 1941 Japan attacked Pearl Harbor and this was a major event that got America directly involved in WW2." But then on the test we'd see questions such as "What important event happened December 7th 1941?", "What was the date of the Pearl Harbor attack?", "What major event led to the US becoming a direct player in the war and when did this event occur?" and so on. Granted, because we do such testing I'd expect GPT to be able to reasonably answer many of these same questions but I'm very much under the impression that humans are far more robust/generalized and would quickly learn these reversals. And to stress, conditioned on knowing the forward direction to a strong degree. If you're fuzzy about the forward direction I'm not surprised if the reversal is even more fuzzy. Definitely biased. But more importantly, I do not think we have the same strong algorithmic bias in our encoding methods that transformers have. That's my main claim, just to clarify.
But I'm conjecturing a lot here. I think these are reasonable and we have some strong evidence for this. Maybe this is more situational than I'm presuming and I'm definitely working off of biased sampling. So discussion helps refine the ideas.
I think anyone who's used Anki to learn anything will tell you reversal is by no means free.
고양이 on the front deck, Cat on the back.
You can see this direction and remember cat is the correct answer every single time but completely blank in the other direction. Cat on the front. What Korean word is at the back ?
Conditions I mean: 고양이 => <concept of cat>; cat => <concept of cat>; either [고양이 => cat || cat => 고양이]. Under these conditions 고양이 == cat && cat == 고양이 should be known.
Your conditions: 고양이 ~> <concept of cat>; cat => <concept of cat>; [ 고양이 -> cat && cat ~> ...고양이?] (or something, trying to express you're preferentially learned one direction, usually mapping back to mother tongue while learning)
By no means are we strictly invariant. But I'd say that a good measure of your skills are in fact to be invariant. But also like you are pointing out, we also kinda train ourselves that way because it creates stronger pathways. Training methods are important too! And augmentation is critical to both human learning and machine learning. They're specifically critical to generalization abilities. You'll also see this in humans. I notice test focused educations tend (there's no "always") to create less generalization of concepts as you're over learning a thing presented in a specific way rather than focusing on the concept as a whole. Form of metric hacking if you will.
Side note: if you have tips for an extreme beginner learning Korean I'd really appreciate them. Trying to reduce the burden on my gf having to translate everything for me when we're around other Koreans.
Anyway, best of luck.