When they're free, it's pretty cool. But charge an amount where there's actual profit in the product? Suddenly seems very expensive and not economically viable for a lot of use cases.
We are still in the "you need a supercomputer" phase of these models for now. Something like DALLE mini is much more accessible but the results aren't good enough. Early early days.
They really aren't that large by the contemporary scaling race standards. DALLE-2 has 3.5B parameters, which should fit on an old GPU like Nvidia RTX2080, especially if you optimize your model for inference [1][2] which is commonly done by ML engineers to minimize costs. With optimized model, your memory footprint is ~1 byte per parameter, and some less than 1 ratio (commonly ~0.2) of all parameters to store intermediate activations.
You should be able to run it on Apple M1/M2 with 16GB RAM via CoreML pretty fine, if an order of magnitude slower than on an A100.
Training isn't unreasonably costly as well: you can train a model given O(100k)$ which is less than a yearly salary of a mid-tier developer in silicon valley.
There is no reason these models shouldn't be trained cooperatively and run locally on our own machines. If someone is interested in cooperating with me on such a project, my email is in the profile.
1. https://arxiv.org/abs/2206.01861
2. https://pytorch.org/blog/introduction-to-quantization-on-pyt...
Still, I think optimization of diffusion models for efficient inference isn't yet pushed to the limits. At least if we look at what's available to the public - AFAIK public inference software distributions didn't even quantize their weights.
Excet we’re already waiting 90+ seconds for dall-e mini.
What are the resources needed to train this model?
If someone just gave you the model for free, what resources would you need to use it to generate new results?
This means the parameters of the trained model fit in something like 7GB (decoder only, half-precision floats) to 24GB (full model, full-precision). To actually run the model, you will need to store those parameters, as well as the activations for each parameter on each image you are running, in (video) memory. To run the full model on device at inference time (rather than r/w to host between each stage of the model) you would probably want an enterprise cloud/data-center GPU like an NVIDIA A100, especially if running batches of more than one image.
The training set size is ~97TB of imagery. I don't think they've shared exactly how long the model trained for, but the original CLIP dataset announcement used some benchmark GPU training tasks that were 16 GPU-days each. If I were to WAG the training time for their commercial DALL-E 2 model, it'd probably be a couple of weeks of training distributed across a couple hundred GPUs. For better insight into what it takes to train (the different stages/components of) a comparable model, you can look through an open-source effort to replicate DALL-E 2.[2]
[0] https://cdn.openai.com/papers/dall-e-2.pdf [1] https://openai.com/blog/clip/ [2] https://github.com/lucidrains/dalle2-pytorch
> you would probably want an enterprise cloud/data-center GPU like an NVIDIA A100, especially if running batches of more than one image.
That doesn't seem so bad.
looks up price of NVIDIA A100 - $20,000
oh...ok I'll probably just pay for the service then
For the half-precision version at 7GB there are a ton more options (the RTX 3060 has 12GB for example at ~$450).
I do hope that the conversation starts to acknowledge the difference between sunk costs and running costs.
Employees, office leases and equiment are all happening, regardless and ongoing.
Training DALL-E 2: very expensive, but done now. A sunk cost where every dollar coming in makes the whole endeavor more profitable.
Operating the trained model: still expensive, but you can chart out exactly how expensive by factoring in hardware and electricity.
I believe that by not explicitly separating these different columns when discussing expense vs profit, we're making it harder than it needs to be to reason about what it actually costs every time someone clicks Generate.
But I seem to remember they were running 1,000+ 32gb GPUs for 3 months to train it and keeping that infrastructure running day-to-day and tweaking parameters as training continued was the bulk of the 100 pages. It is beyond the reach of anybody but a really big company, at least in the area of very large models, and the large models are where all the recent results are. I wish I was more bullish on algorithm improvements meaning you can get better results on less hardware; there will definitely be some algorithm improvements, but I think we might really need more powerful hardware too. Or pooled resources. Something. These models are huge.
Is https://github.com/facebookresearch/metaseq/blob/main/projec... what you're referring to?
Training requires even more GPUs, and I wouldn’t be surprised if they used more than 100 and trained over 3 months.
Based on this blog post where they scale to 7,500 'nodes', they say:
> A large machine learning job spans many nodes and runs most efficiently when it has access to all of the hardware resources on each node.
So I wouldn't be surprised if they do have a total of 7500+ GPUs to balance workloads between. TO add, OpenAI has a long history of getting unlimited access to Google's clusters of GPUs (nowadays they pay for it, though). When they were training 'OpenAI Five' to play Dota 2 at the highest level, they were using 256 P100 GPUs on GCP[0] and they casually threw 256 GPUs at 'clip' for a short while in January of 2021[1].
As for how they do it, see these posts:
https://openai.com/blog/techniques-for-training-large-neural...
https://openai.com/blog/triton/
I’ve been creating generative art since 2016 and I’ve been anxiously waiting for my invite. I wont be able to afford to generate the volume of images it takes to get good ones at this price point.
I can afford $20/mo for something like this but I just can’t swing $200 to $300 it realistically takes to get interesting art out of these CLIP-centric models.
Heck, the initial 50 images isn’t even enough to get the hang of how the model behaves.
Meanwhile we should prepare ourselves for a future where the best generative models cost a lot more as these companies slice and dice the (huge) burgeoning market here.
Majestic diffusion - https://github.com/multimodalart/majesty-diffusion
Centipede diffusion - https://colab.research.google.com/github/Zalring/Centipede_D...
Colab is dogshit if you don't pay
My brother is a digital artist and while excited at first he found it to be not all that useful. Mainly because it falls apart with complex prompts, especially when you have a few people or objects in a scene, or specific details you need represented, or a specific composition. You can do a lot with in-painting but it requires burning a lot of credits.
On the personal side, I've been getting into game development, but the biggest roadblock is creating concept art. I'm an artist but it takes a huge amount of time to get the ideas on paper. Using DALLE will be a massive benefit and will let me expedite that process.
It's important to note that this is not replacing my entire creative process. But it solves the issue I have, where I'm lying in bed imagining a scene in my mind, but don't have the time or energy to sketch it out myself.
this is what I really like about DALLE-mini, it's ability to create pretty good basic outlines for a scene. it's low resolution enough that there's room for your own creativity while giving you a good template to spring off from. things like poses, composition of multiple people, etc.
Generating itself can be art. I’m not going to win a Pulitzer here, it’s for the personal joy of it, but I will certainly never get tired of it.
Knowing that it will only get better - animation cannot be far behind - makes me feel genuinely excited to be alive.
It’s good at “in the style of” but there’s no “in a new style”.
It has a house style too that tends to feel Reddit-like.
Either way, I feel like your view is an exhaustingly pessimistic take on AI-generated art. I mean, sure, most of what DALL-E generates is pretty mundane, but other times I have been surprised at how bizarre and unique certain images are.
You seem to imply that because an AI is not human, its art is not imbued with meaning or originality -- but I find that an AI's non-human nature is precisely what _makes_ the art so original and meaningful.
At the extreme limit, maybe. But within art or even digital art, then new styles are actually not that rare, humans are pretty good at generating them. Maybe they grab inspiration from nature, visual phenomena, etc, so in that sense it's not "new" but it is "new to the medium". In art you new styles all the time. DALL-E will never do that by it's very nature, and so it's easy to see how it's boring.
And that's just the stylistic level, but it's happening at almost all levels. It's almost definitional that it doesn't innovate, only remix.
It's strange framing this as pessimistic, it's not really optimistic nor pessimistic, it just is. It's also not AI, and that's important to realize: it's a statistical model that generates purely based on pre-existing training. It's very nature is without-meaning and without-originality. That doesn't detract from it being cool or interesting or helpful or enjoyable. I find it cool and useful.
But it's not innovative or creative or meaningful by itself.
That's a pretty bold claim. What are humans but statistical models that generate based on pre-existing training -- and yet, are humans not with-meaning and with-originality?
Of course, human brains are some large order of magnitude more complex than the neural nets that underlie most AIs, but we can already see areas where these "simplistic" AIs outperform humans on specific tasks. So what prevents the arts from being one of those areas? If not now, in some not-too-distant future?
One thing humans seem to have that is beyond statistics is creativity. In that stats explain what is, and creativity takes what is and makes a dot outside that other people appreciate. No model has demonstrated even attempting the dot, let alone having a good chance of success. What DALL-E does is draw a dot between a few existing points, but never outside.
Humans incidentally have three more things that make for interestingness: emotions derived from feelings, long term memory, and roughly storytelling (~ an ability to turn long term memory into long form recall with a specific reaction intended to a specific audience). I don’t think ML has any of those, but it likely (eventually) gets the latter two.
Meaningness/interestingness require at least a few of those, and it’s what puts most art in a different category than games or math.
If your task is "show me something that breaks through our hyperspeed media", then I guess some obscure museum is a better place than an ML model.
If your task is "find the best variation on theme X" or "quick draft visualization", they are often very useful. I am sure there will be many further tasks to which current and future models will be well suited. They are not magic picture machines. At least not yet.
I'd disagree.
- One of the first queries I did made some interesting chairs, I would genuinely buy the first if it was sanely priced: https://www.ryanmercer.com/ryansthoughts/2022/6/17/dall-e-2-...
- One of the first H R Giger inspired works I did made some really interesting computers, I would (and may) hang some of these https://www.ryanmercer.com/ryansthoughts/2022/6/19/dall-e-2-...
- The first wood carving query I did generated solid gold of Vikings eating pizza, this is the kind of thing I'd see in a hole in the wall restaurant and absolutely love https://www.ryanmercer.com/ryansthoughts/2022/6/17/dall-e-2-...
- The first "painting" in this series I may very well print and hang in my office https://www.ryanmercer.com/ryansthoughts/2022/6/27/dall-e-2-...
- These H.R. Giger chairs 100% look like something I'd expect to see in a modern art museum https://www.ryanmercer.com/ryansthoughts/2022/6/24/dall-e-2-...
- If you want to create new characters for something, and are lacking inspiration, I think it could be extremely useful to artists. For example these variations of Don't Hug Me I'm Scared https://www.ryanmercer.com/ryansthoughts/2022/6/21/dall-e-2-...
I've got thousands of queries, and a LOT of them have generated things I genuinely see as having artistic value, I've probably got 200~ images that I would 100% hang/display in my home ('woodcarving' and 'stone carving' queries rarely disappointed me)
is it some unique form of art, no, but can it produce works I want to see in a medium or style that already exists to a level that it is believable as authentic human art, absolutely.
People like me, with zero artistic ability, are able to take part in creating visually pleasing works. I imagine artists would also find great value in it, being able to feed a few queries in with what they are thinking of creating to draw inspiration, or even putting their own work in and generating variations that may lead to inspiration for new works.
I'm not going to try and profit from the images, I don't need them for any business uses or anything, so to me it was a fun for a while and now just something I'll largely put out of mind.
directly from the website
Multimodal.art (https://multimodal.art/) is working on a free version of something like DALLE, though it's not that good as of yet.
This belongs on /r/linkedinlunatics
Wait until the next edition comes out where it automatically learns the sorts of things that crack you up and starts generating them without any input from you.
It'll use hideous amounts of compute.
If you get a large percentage hooked on TikTok you can change and undermine democracy.
Starting to believe representative democracy and social media are incompatible.
In fact, I've been glad to have a 50/day limit, because it helps me contain my hyperfocus instincts.
The information about new pricing is, to me as someone just enjoying making crazy imagines, a huge drag. It means that to do the same 50/day I'd be spending $300/month.
OpenAI: introduce a $20/month non-commercial plan for 50/day, and I'll be at the front of the line.