It also goes against common sense. Math, reading, and so many things are all learned, but frequently used by people who have a high IQ. Are we to assume that these things are not learned, because the author thinks IQ is the cause of reading skill, math, and similar things?
I've bothered going through the labor to learn the skill. I've seen my baseline measurement of recall ability improved by the training time I invested in learning the skill. I haven't bothered to get to a competitive level, but I don't need to do so to be certain that humans can learn skills by practicing them with the intention to improve.
The skeptic is just wrong. People can learn to improve their memory.
Critically, that doesn't mean that the skills aren't learnable. It means that people with the most advantages can exploit those advantages to gain a greater deal of advantage than someone who has less advantage.
It's been a while, but I have a recollection of Foer going into the question of why more people don't do this in his book. It mostly has to do with the amount of time one needs to commit to practice, coupled with limited transferability of this skill.
To give another anecdote, I've worked with a very high-performing engineer who spent a bunch of his free time building memory palaces for new codebases to ensure retention. I'm sure he was above average IQ-wise, but his productivity likely benefited from his approach as much as from his innate abilities.