> we find intricate structure at every scale
> At every length scale, small changes in the hyperparameters can lead to large changes in training dynamics
This is the definition of Chaos though no? Butterfly flaps its wings, hurricane on other side of the planet...
I recall similar tests being done on paintings by Pollock and of other artists trying to copy his style to determine authenticity[3].
[1]: https://en.wikipedia.org/wiki/Fractal_dimension
[2]: https://mathworld.wolfram.com/Fractal.html
[3]: https://cpb-us-e1.wpmucdn.com/blogs.uoregon.edu/dist/e/12535...
I think it probably stems from the fact that all the fractals, I have seen, for which the dimension can be analytically calculated do show obvious patterns of similarity at different scales.
They point out that the usual measure of fractal dimension, or capacity dimension[2], doesn't consider the physical size of the features and can thus be inaccurate. Instead they suggest using the information dimension[3], which is bounded by the capacity dimension.
[1]: https://doi.org/10.1016/j.chaos.2018.05.008 (full text available on that hub of science)
[2]: https://mathworld.wolfram.com/CapacityDimension.html
[3]: https://mathworld.wolfram.com/InformationDimension.html
https://medium.com/@h.a.papageorgiou/the-reality-of-the-ruli...