When you train a neural net for Donkeycar with camera images plus the joystick commands of the driver, isn't that close to nerve impulses already?

> I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics

Neuralink. Musk warning though.

For reference, see Neuralink Launch Event at 59:33 [0], and continue watching through until Musk takes over again. The technical information there is highly relevant to a multi-modal AI model with sensory input/output.

https://youtu.be/r-vbh3t7WVI?t=3575

There are a few folks working on this in neuroscience, e.g. training transformers to "decode" neural activity (https://arxiv.org/abs/2310.16046). It's still pretty new and a bit unclear what the most promising path forward is, but will be interesting to see where things go. One challenge that gets brought up a lot is that neuroscience data is often high-dimensional and with limited samples (since it's traditionally been quite expensive to record neurons for extended periods), which is a fairly different regime from the very large data sets typically used to train LLMs, etc.

There are ‘spiking neural networks’ that operate in a manner that more closely emulates how neurons communicate. One idea I think that is interesting to think about is that we build a neural network that operates in a way that is effectively ‘native’ to our mind, so it’s less like there’s a hidden keyboard and screen in your brain, but that it simply becomes new space you can explore in your mind.

Or learn king fu.

Like Cortical labs? Neurons integrated on a silicon chip https://corticallabs.com/cl1.html