Predict the next frame. Results of training a transformer on "visual sentences"

Re:Predict the next frame. Results of training a transformer... by raidicy on Tuesday December 05, 2023 @12:00PM (#38533522)

These sentiments are pretty close to my own. I read a paper that claimed that llms are General Pattern Machines and could be used to complete small games in gym environments. It seems to me that if these things really are General Pattern Machines all we have to do is figure out a way to represent any data as a pattern and try and predict the next step in the pattern right?

The multi token[1] project which allows you to take any type of data and turn it into a token it's pretty interesting and seems like it's going in this direction.

I would really like to see a framework where you can take any modality of any type turn it into a series of tokens and just cram it into a language model and effectively turning into a multimodal model with almost no effort.

[0]https://general-pattern-machines.github.io/ [1] https://github.com/sshh12/multi_token

[ Parent | on HN ]

Re:Predict the next frame. Results of training a transformer... by kelseyfrog on Tuesday December 05, 2023 @04:24PM (#38537095)

If they're looking for a name, The Glass Bead Game isn't a bad start.

[ Parent | on HN ]

Re:Predict the next frame. Results of training a transformer... by texas2toss on Tuesday December 05, 2023 @06:06PM (#38538190)

Probably my favorite “smart” book of all time

[ Parent | on HN ]

Re:Predict the next frame. Results of training a transformer... by sshh12 on Tuesday December 05, 2023 @07:47PM (#38539066)

Yeah this is exactly what I was thinking when building multi_token! Just bottlenecked so far by my own GPU, datasets, and freetime

[ Parent | on HN ]

Re:Predict the next frame. Results of training a transformer... by raidicy on Wednesday December 06, 2023 @07:11AM (#38542813)

Thank you thank you thank you! I think it's a wonderful repo it's a shame I haven't been able to dig deeper into it. I starred your project as soon as I saw it!

[ Parent | on HN ]