Re:Imagen, a text-to-image diffusion model
by Mockapapella on Monday May 23, 2022 @05:27PM
(#31484960)
Transformers are parallelize-able, right? What’s stopping a large group of people from pooling their compute power together and working towards something like this? IIRC there were some crypto projects a while back that we’re trying to create something similar (golem?)
[ Parent
| on HN ]
There are people working on reproducing the models, see here for Dall-E 2 for example: https://github.com/lucidrains/DALLE2-pytorch
It's often not worth it to decentralize the computation of the trained model though but it's not hard to get donated cycles and groups are working on it. Don't fret because Google isn't releasing the API/code. They released the paper and that's all you need.
There are the Eleuther.ai and BigScience projects working on public foundation models. They have a few releases already and currently training GPT-3 sized models.
Re:Imagen, a text-to-image diffusion model
by sinenomine on Monday May 23, 2022 @09:55PM
(#31487182)
You really need a decent infiniband-linked cluster to train large models.