Re:Meta AI Unleashes Megabyte, a Revolutionary Scalable Mode...
by DrScientist on Friday June 02, 2023 @11:21AM
(#36166370)
> But they can't fire 3B times a second.
Fair point - but I'd argue the power comes for the combinatoric complexity, not the speed of operation.
So those timing effects can also combine with the simple connections - so if a neuron has 2 connections - the output can be controlled by not just the last signals for the 2 connections, but the relative timing of each of them.
Suddenly your input space you are operating over is much much bigger - you've got 2 inputs and hundreds of ways the timing could be adjusted. Massively bigger.
The fact that the whole thing doesn't operate in a big synchronous cycle is a huge jump in possible states.
Fair point - but I'd argue the power comes for the combinatoric complexity, not the speed of operation.
So those timing effects can also combine with the simple connections - so if a neuron has 2 connections - the output can be controlled by not just the last signals for the 2 connections, but the relative timing of each of them.
Suddenly your input space you are operating over is much much bigger - you've got 2 inputs and hundreds of ways the timing could be adjusted. Massively bigger.
The fact that the whole thing doesn't operate in a big synchronous cycle is a huge jump in possible states.