Therefore it shouldn't be surprising that that you can find probability distributions that share the first two moments, but differ wildly in the higher moments (skewness, kurtosis) or even the number of modes (bimodal, trimodal, etc.).
Not everything in the world is Gaussian, but we knew that.
The paper is actually really interesting, as it provides a useful insight into what summary statistics mean and the importance of data visualization.
The second part of the article makes a claim that is worth discussing -- that probabilistic inference does not work well in nonlinear domains.
I think this is true in one sense, but in many practical situations we can usually to get something to work well enough for it to be useful. Assuming we can (approximately) split up the domain into multiple regions and get probability distributions for each region (with transition functions between them), we can continue to make inferences under nonlinearity. That said, this is a concession. Perhaps there are techniques in deep learning that more general in nature.
I want to dig deeper into this but two things come to mind:
1. George Sugihara's and Floris Takens' works on quantifying causation in non-linear systems. Correlation doesn't imply causation, so why not look at causation? Here's a writeup on it: https://www.quantamagazine.org/chaos-theory-in-ecology-predi...
2. Andrei Khrennikov argues that quantum models can be expressed with Kolmogorovian (classical) probability theory. He points out that many probability theorists have glossed over an entire subtlety of Kolmogorov's work: that probability spaces need not be finite dimensional, that they can be subspaces of higher dimensional, even infinite dimensional probability spaces. Khrennikov's philosophical convictions are not widely shared by quantum theorists, but his math is solid (he seems to be in pursuit of quantum theories with "hidden variables", like Einstein was looking for). http://www.sciencedirect.com/science/article/pii/S0375960103...
Even in the gif shown in this article, you might suppose that each point set could be modeled in higher dimensional terms, for example adding some kind of discrete curl-like measure. But even without going there: something as simple as a changing to a polar basis should reveal different statistics.
And isn't this kind of what deep learning is all about? Going between higher and lower dimensional representations?