Oh yeah. The exciting thing is that is pretty low hanging fruit (at least for the common modalities)

What would a Large Language Model that can manipulate audio-visual data as expertly as it can manipulate text look like ? This is beyond just Text to Speech or Captioning and Image Q&A. I think we'll find out very soon.

Yeah, I think this is now in the "we know it can be done, and how to do it". And also som

I think we might have audio2audio editing on the level of Stable Diffusion within a year or two. Based on recent progress with AudioLM etc.

For short clips, we will probably get video2video editing in the same timeframe. While it is computationally more challenging, it might end up being here before audio editing - because video is so popular in todays social media and marketing landscape.

Usable joint video and audio editing (that is coherent) will probably take longer.

These sentiments are pretty close to my own. I read a paper that claimed that llms are General Pattern Machines and could be used to complete small games in gym environments. It seems to me that if these things really are General Pattern Machines all we have to do is figure out a way to represent any data as a pattern and try and predict the next step in the pattern right?

The multi token[1] project which allows you to take any type of data and turn it into a token it's pretty interesting and seems like it's going in this direction.

I would really like to see a framework where you can take any modality of any type turn it into a series of tokens and just cram it into a language model and effectively turning into a multimodal model with almost no effort.

[0]https://general-pattern-machines.github.io/ [1] https://github.com/sshh12/multi_token

If they're looking for a name, The Glass Bead Game isn't a bad start.

Probably my favorite “smart” book of all time

Yeah this is exactly what I was thinking when building multi_token! Just bottlenecked so far by my own GPU, datasets, and freetime

Thank you thank you thank you! I think it's a wonderful repo it's a shame I haven't been able to dig deeper into it. I starred your project as soon as I saw it!

across a bunch of different gradients. senses will be the next step, humans grok that good and easy. once multiple gradients are considered, non-sensory gradients are going to be next.

this is all a bunch of gobbledygook until it isn’t.

That pretty much already exists. Look at DeepMind's Gato: all tasks and modalities are simply sequences of tokens, everything from 'predict English text' to 'predict VAE image token sequences' to 'predict robotic arms commands and movements IRL'.

Ah, yes, I'd forgotten about Gato. Thank you for reminding me. There's so much research activity that the Gato paper feels as if it was published eons ago. There's only so much I can retain in my puny little human mind at once!

In any case, I'm not sure Gato qualifies as a "large" model with 1.2B parameters -- it's kinda right below the threshold at which it could or would start exhibiting emergent behaviors. Maybe a new Gato with 10's or 100's of billions of parameters operating in the physical world?

Yes. Gato was a good proof-of-concept that the Decision Transformer approach of 'just model literally everything as a sequence' scales well and doesn't exhibit some sort of catastrophic interference and can successfully imitation-learn from all the expert datasets, and a bit of transfer. But they need to push it at least another OOM or 2 to show major transfer, some emergences, and ideally do both from-scratch learning and additional learning on many new tasks. We continue to wait. :(

I hope it didn't all get rolled up into Gemini and become a state secret they'll never publish on again, or lost in the shuffle in the chaos of the DeepMind/Brain merger/liquidation.

> ...lost in the shuffle in the chaos of the DeepMind/Brain merger/liquidation

That's the most likely explanation, in my view.

GATO was heavily biased towards tasks in a simple simulator, but didn’t exhibit emergent behaviours

Yeah its going to happen. We will see and speak with intelligent machines

I did that a few years back: https://arxiv.org/abs/2011.11751

With a large model? How many parameters?

See my other comment here:

https://news.ycombinator.com/item?id=38536178

A couple millions IIRC. Nothing "large" compared to modern transformer models.

Thanks for getting back to me. That's what I thought. The magic seems to start happening in the low billions of parameters -- and I say "seems" b/c there's no consensus as to whether it's really truly magic! In any case, it's a shame that most of the human brainpower capable of improving SotA AI doesn't have access to large-scale resources.

One can argue there are ~8bln of these robots already roaming the Earth.

Let me guess. Someone will train a LLM on the stock prices to predict the stock market. It might work as well as how human has predicted the market.

Past performance is not a guarantee of future results.

Yepp. I think a robot who can easily follow your commands should be doable in 10-20 years.

I plan to buy a farm when I have the money and I'm pretty sure while I will/want to do a lot of hands on renovation and sculpting (park, etc) long term some type of robot should be good and affordable enough to take over when I'm too old.