Do vision transformers see like convolutional neural networks?

Re:Do vision transformers see like convolutional neural netw... by mrfusion on Wednesday August 25, 2021 @12:26PM (#28303611)

Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting results.

[ Parent | on HN ]

Re:Do vision transformers see like convolutional neural netw... by sillysaurusx on Wednesday August 25, 2021 @12:28PM (#28303636)

https://github.com/openai/CLIP

The results are quite interesting:

https://www.reddit.com/r/Art/comments/p866wv/deep_dive_meai_...

[ Parent | on HN ]

Re:Do vision transformers see like convolutional neural netw... by codetrotter on Wednesday August 25, 2021 @04:50PM (#28306779)

And here is a short guide and a link to a Google Collab notebook that anyone can use to create their own AI-powered art using VQGAN+CLIP: https://sourceful.us/doc/935/introduction-to-vqganclip

[ Parent | on HN ]

Re:Do vision transformers see like convolutional neural netw... by abalaji on Wednesday August 25, 2021 @12:28PM (#28303634)

yeah there has definitely been work done in that space: it’s called multi-modal models

not sure if this is the latest work but here’s some results from Google’s AI Blog

https://ai.googleblog.com/2017/06/multimodel-multi-task-mach...

[ Parent | on HN ]

Re:Do vision transformers see like convolutional neural netw... by im3w1l on Wednesday August 25, 2021 @02:49PM (#28305393)

What would be really cool is neural networks with routing. Like circuit switching or packet switching. No idea how you would train such a beast though.

Like imagine the vision part making a phonecall to the natural language part to ask it for help with something.

[ Parent | on HN ]

Re:Do vision transformers see like convolutional neural netw... by TylerLives on Wednesday August 25, 2021 @03:35PM (#28305904)

Sounds like The Society of Mind - https://en.m.wikipedia.org/wiki/Society_of_Mind

[ Parent | on HN ]

Re:Do vision transformers see like convolutional neural netw... by wittenator on Wednesday August 25, 2021 @04:57PM (#28306886)

Capsule networks have a routing algorithm as far as I know

[ Parent | on HN ]