There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.
The reason I like the CPU approach is the memory scaling is bonkers compared to GPU. You can buy a server that has 12TB of DRAM (in stock right now) for the cost of 1 of those H100 GPU systems. This is enough memory to hold over 3 trillion parameters with full 32-bit FP resolution. Employ some downsampling and you could get even more ridiculous.
If 12TB isn't enough, you can always reach for things like RDMA and high speed interconnects. You could probably get 100 trillion parameters into 1 rack. At some point you'll need to add hierarchy to the SNN so that multiple racks & datacenters can work together.
Imagine the power savings... It's not exactly a walk in the park, but those DIMMs are very eco friendly compared to GPUs. You don't need a whole lot of CPU cores in my proposal either. 8-16 very fast cores per box would probably be more than enough, looking at how fintech does things. 1 thread is actually running the entire show in my current prototype. The other threads are for spike timers & managing other external signals.
Right now I'm building my homelab server which aimed to fit 1 TB RAM and 2 CPUs with ~100 cores total.
It will cost like 0.1% of what I need to pay for GPU cluster with the same memory size :)
This presupposes we've explored this space thoroughly, and we haven't. When everything you do with NNs improves results (how it mostly is now), that means not enough people are trying out ideas and new things.
I don't think you can invoke EMH-like reasoning quite yet. Give us a nice long winter like physics has had, and then we can use this heuristic.
There's a hypothesis in the parent comment - better handling of the time domain will lead to better modeling - which is actually fairly independent of architecture. So, there's going to be a number of possible ways to build better time modeling, ranging from tweaks to existing architecture to completely rebuilding Rome. So, if better time modeling really is a limitation, you don't need to rebuild Rome to find out.
In fact, I might argue that S4 layers already provide this improved time handling in the current world, and is proving very successful, which would again widen the moat for SNNs.
Bad, BAD idea. Remember the Tay chatbot, which 4chan managed to turn into a raging Nazi in the matter of a few hours?
Spiking networks also lend themselves nicely to some elegant learning rules, such as STDP. Being able to perform unsupervised learning at the grain of each action potential is really important in my mind. This gives you all kinds of ridiculous capabilities, most notably being the ability to train the model while it's live in production (learning & use are effectively the same thing).
These networks also provide a sort of deterministic, event-over-time tracing that is absent in the models we see today. In my prototypes, the action potentials are serialized through a ring buffer, and then logged off to a database in order to perfectly replay any given session. This information can be used to bootstrap the model (offline training) by "rewinding" things very precisely and otherwise branching time to your advantage.
The #1 reason I've been thinking about this path is that low-latency, serialized, real-time signal processing is somewhat antagonistic to GPU acceleration. I fear there is an appreciable % of AI research predicated on some notion that you need at least 1 beefy GPU to start doing your work. Looking at fintech, we are able to discover some very interesting pieces of technology which can service streams of events at unbelievable rates and scales - and they only depend on a handful of CPU cores in order to achieve this.
Right now, I think A Time Domain Is All You Need. I was inspired to go outside of the box by this paper: https://arxiv.org/abs/2304.06035. Part 11 got me thinking.