That may sound surprising, but it's actually not unusual. It's worth remembering that the difference between highly local encoding and everything encoding everything at the same time, is often a matter of a reversible transformation.
For example, take FFT of an image, and the result will be an image where each pixel encodes information about all pixels of the original image simultaneously. And Fourier transform - a shift from a time/spatial domain to frequency domain - is quite simple, very useful, and occurs in nature.
Of course for any given location this wraps around with a period of the least common multiple between the grid cells' periods (I think? Had to ask Bing about that one, he was helpful for once). This is why if you try to map locations you get a hypertorus.
I think the fourier transform might be a really approachable version because it is already in so much signal processing. Sometimes its periodicity can work against you.
From what I can tell you, what you need are:
1. invertible, identifiable basis
2. Something like Parseval's Identity
3. Able to "smooth" out things like the heaviside function or dirac deltas
4. Workable on graphs as well
5. decays at infinity
6. eigenfunctions are all orthonormal
Turns out, making something like that gets you something similar to the fourier transform, and the math is probably the simplest since you are working with easy exponentials that work nicely with convolution.
People have done similar work with Chebyshev polynomials, and there are actually a lot of applications in ML/AI using Chebyshev polynomials in graphical neural networks and triangulation.
edit:
The fourier transform also comes about pretty nicely in quantum mechanics because of poisson bracket stuff and its nice derivative properties for position-momentum space. I am don't think other transforms/function basis come out so naturally.
Nobel disease, but even without the Nobel-price?
Topology is hard for me, visualising multi-dimensional topological manifolds even more so, however, I am intrigued by the opportunities, we use graphs and other forms of visualisation to reveal topology, and it would seem reasonable that behaviours of NN that are 'transferrable' e.g.. trained on one set of data and able to operate on another category of data, may be a 'shape', a complicated one, but perhaps one that would once revealed would aid in the understanding of what happens under the hood.
Over that time, brain function was mostly concerned with aiding the organism to find food, grow, reproduce, and avoid being eaten, rather than language, logic, mathematics, arts, and so forth. Its rather astonishing that humans are somehow now able to do the latter, using brains evolved to do the former.
We appear to have a capacity for substantially greater sophsitication in those domains, but none are unique to us except when we artificially define them to be. Remember that words like "language", "logic", "art", etc are cultural inventions with a fuzzy and fluid relationship to whatever real-word "stuff" they refer to, not natural kinds that themselves have sharp and perennial definitions.
Unless you choose to define the word as that which only humans can acheive, a spider's web elegently reflects "mathematics" just as much as some beautiful proof in set theory; a conflicted bird debating itself over which stem to use in its nest reflects artistic attention just as as a painter choosing their next color; a cat chirping or mewing or yowling reflects language just as me writing this comment.
The sophistication doesn't go as far, by our eye at least, in any of these animal examples, and so we don't expect the spider to confirm Fermat's Last Theorem or the bird to feature their nest in a gallery (actually...) or a cat to compose formal poetry, but the essential bits that we extend with our sophistication are all ancient and widespread throughout nature.
It's still astonishing that any life can do so many of the things it but I guess that's apparently what billions of years of "pretraining" on unfathomably efficient machines gets you.
Incidentally, it's wild to see people believe that a stream of fmults pushing through a trillion transistors would get you even close to the sophistication of any of life's intelligence. For current-AI-skeptical materialists, it's usually not a doubt about whether silicon and software might conceivably be intelligent, but it can just seem absurd to believe the grossly crude and narrow innovations of recent years are even close. You need to have a very shallow, narrow, almost willfully blinded, appreciation of the "intelligence" exhibited throughout all biological life to think that you unlocked the silicon version of it all in a pretty-good chatbot running on Azure.
[0] https://blog.cambridgecoaching.com/ants-go-marching-fun-fact....
From https://news.ycombinator.com/item?id=38334538#38336861 :
> Which NN architectures could be sufficient to simulate the entire human brain with spreading activation in 11 dimensions?
Does it? My read is that biological neutral architecture is a consequence of biological constraints that artificial NNs mostly don't face, so we don't necessarily need to try to copy it.
A fair chunk of AI work boils down to “make something that acts like a human.” On the other end of the spectrum is stuff that is more specialized, like very targeted classifiers; there is no reason to expect those would benefit from this.
We’re just normalizing to our innate sensibilities. We have no idea if we’re making intelligence or what that even means as us humans must work within the constraints we evolved into. We have no idea if we generalized consciousness as it could exist across space time.
You and I will never exist outside our universe and observe what makes it tick. We’re hanging out on Earth making mannequins talk, hallucinating we’re gods because of it. Humans are artificial intelligence given their lack of direct observation of so much of the universe.
Conway came up with his game of life after the universe. Sorry, AI researchers, life, consciousness and visualization were already created by reality. We’re just working on an easy to use Dewey Decimal system to catalog it.
personally i absolutely do think that for generating convincingly human-like intelligence you also need some human constraints, otherwise you will get some uncanny valley.
another example would be alpha-zeros play style. AIs don't play like humans, they maximize their chance of winning in the long term without going for good looking opportunities that hurt their chances in the long run (like human players do).
I guess I just find the goal of imitating human intelligence including all its mistakes to be a silly goal. The only time you want that instead of an actual human is if you're trying to deceive people into thinking your AI is a human. Otherwise, you just want the correct answer (or, if you're afraid, you want a strictly sub-human intelligence).
We've developed numerous different learning algorithms that are biologically plausible, but they all kinda work like backpropagation but worse, so we stuck with backpropagation. We've made more complicated neurons that better resemble biological neurons, but it is faster and works better if you just add extra simple neurons, so we do that instead. Spiking neural networks have connection patterns more similar to what you see in the brain, but they learn slower and are tougher to work with than regular layered neural networks, so we use layered neural networks instead.
Biology is Biology and Silicon is Silicon. Sometimes, Constraints can be just that - Constraints and not some secret sauce.
The work suggests that existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard.