As a complete layman reading that paper, Skilling's arguments seem quite damning of IG. In particular, Amari's fundamental assumption seems to be refuted:
"For inference, the only acceptable value for the Rényi-Tsallis parameter is α = 1, which is the correct information. That negates the generalisation to α != 1 which underlies Amari’s “α-divergences” in information geometry."
Have any IG proponents responded to or refuted Skilling's critiques? This is interesting because Shun'ichi Amari is credited amongst others with advancing the field in the 80's[1].
http://djafari.free.fr/MaxEnt2014/papers/Tutorial4_paper.pdf
The main point is that KL-divergence is not a metric, so imagining it as a distance in a space may give you some wrong intuitions. Its matrix of 2nd derivatives, the Fischer Information, works as a local metric, but then many people want to draw global pictures that try to extend this back to a global metric, which doesn't actually work.