Open source efforts to produce a similar model seem to have failed due to the extreme compute requirements for finetuning. For example, Waifu Diffusion using 8XA40[0] have not managed to bend SDXL to their will after potentially months of training.
If you need 256xH100 to even finetune the model for your use case, what's stopping you from just training your own base model? Not much, as it turns out. Developers of NovelAI have stated they'll train from scratch for the next version of their model a few weeks ago.
So I agree, even with the licensing changes things might be looking somewhat dire for SAI.
https://gist.github.com/harubaru/f727cedacae336d1f7877c4bbe2...
I have not heard about the team upgrading or downgrading from the hardware mentioned there, so I assumed it's still the same hardware they use.
>SD XL just came out a month or two ago
About 4.5 months actually.
For the SDXL cannot be finetuned efficiently claim, an attempt at a finetune was released here: https://huggingface.co/hakurei/waifu-diffusion-xl
The team was given early access by StabilityAI to SDXL0.9 for this. You'll have to test it out for yourself, if you're interested in comparing. From my experience, it is a world of a difference between the NovelAI and WaifuDiffusion models in both quality and prompt understanding.
Note, the very baseline I set for the WaifuDiffusionSDXL model was to beat their SD2.1 based model[0], which it did not in my opinion.
SDXL0.9 may have been a worse starting point than SDXL1.0, and certainly there's been more time put in and experience developed finetuning SDXL in the time since WD trained against SDXL0.9 by the people who have released the huge pile of finetunes that have been released since.
A distributed computing project similar to SETI @ Moon wouldn't help with training?
Or are you looking at some very specific definition / threshold for fine tuning here?
When I use a decent paid service, pretty much every prompt gives me a good response out of the box. Which is good, because otherwise I'd have no use for paid services, since I can run it all locally. This causes me to go to a paid service whenever I want something quick, but don't need full control. When I do want full control, I stick to my local solution, but that takes a lot more time.
I'd describe them like "a bike with no handlebars" because they are incredibly difficult to steer to where you want.
For example if you look at the preview images like this one: https://civitai.com/images/3615715
The model seems to have completely ignored a good 35% of the text input, most egregiously I find the (flat chest:2.0), the parenthesis denoting a strengthening of that specific part of the prompt. The values I see people use with good general models range from 1.05~1.15. 2.0 in comparison is an extremely large value, that ended up _still not working at all_, if you take a look at the actual image.
> most egregiously I find the (flat chest:2.0)
The flat chest is fighting to compensate the also heavily weighted (hands on breasts:1.5) which not only affects hand placement but also the concept of "breasts", and the biases trained into many of the community models with that term mean that having that concept in the prompt and heavily weighted takes a lot to counteract. So, no, I don't think its ignoring that.
I've been surprised at the explosion of porn. Well, not actually. Automatic1111 made that easy and anyone that CivitAI knows all too well what those models are being used for. I mean when you give teenagers the ability to undress their crushes[0] what do you think is going to happen (do laws adequately protect people (kids)? Can they? Will this force a shift towards actually chasing producers, distributors, and diddlers?)?
Porn is clearly an in demand market. But what does surprise me is that there's been a lot of work in depth maps and 3D rendering from 2D images in the past few years so I'm a bit surprised that given how popular VR headsets are (according to today's LTT episode, half as many Meta Quest 2s as PS5s have been sold?!). I mean if VR headsets are actually that prolific it seems like there'd be a good market for even just turning a bunch of videos into VR videos, not to mention porn (I don't have a VR headset, but I hear a lot of porn is watched. No Linus, I'm not going to buy second hand...). I think all it takes is for some group to optimize these models like they have for LLaMA and SD (because as a researcher I can sure tell you, we're not the optimizers. Use our work as ballpark figures (e.g. GANs 10x Diffusion) but there's a lot of performance on the table). You could definitely convert video frames to 3D on prosumer grade hardware (say a 90 minute movie in <8hrs? Aka: while you sleep).
There are a lot of wild things that I think AI is going to change that I'm not sure people are really considering (average people anyways or at least stuff that's not making it into popular conversation). ResNet-50 is still probably the most used model btw. Not sure why, but just about every project I see that's not diffusion of an LLM is using this as a backbone despite research models that are smaller, faster, and better (at least on ImageNet-22k and COCO).
(Note the links to various SD compilers).
But the whole field is moving so fast that people aren't even adopting the compilers and optimized implementations at large.
But I wasn't thinking diffusion. Models are big and slow. GANs still reign in terms of speed and model sizes. I mean the StyleGAN-T model is 75M params (lightweight) or 1bn (full) (with 123M for text). That paper notes that the 56 images they use in the Fig 2 takes 6 seconds on a 3090 at 512 resolution. I have a 3080Ti and I can tell you that's about how long it takes for me to generate a batch size of 4 with an optimized TensorRT model. That's a big difference, especially considering those are done with interpolations. I mean the GAN vs Diffusion debate is often a little silly as realistically it is more a matter of application. I'll take diffusion in my photoshop but I'll take StyleGAN for my real time video upscaling.
But yes, I do understand how fast the field is moving. You can check my comment history to verify if register isn't sufficient indication.
It's extremely complicated. Actual CSAM is very illegal, and for good reason. However, artistic depictions of such are... protected 1st Amendment expression[0]. So there's an argument - and I really hate that I'm even saying this - that AI generated CSAM is not prosecutable, as if the law works on SCP-096 rules or something. Furthermore, that's just a subset of all revenge porn, itself a subset of nonconsensual porn. In the US, there's no specific law banning this behavior unless children are involved. The EU doesn't have one either. A specific law targeted at nonconsensual porn is drastically needed, but people keep failing to draft one that isn't either a generalized censorship device or a damp squib.
You can cobble together other laws to target specific behavior - for example, there was a wave of women in the US copyrighting their nudes so they could file DMCA 512 takedown requests at Facebook. But that's got problems - first off, you have to put your nudes in the Library of Congress, which is an own goal; and it only works for revenge porn that the (adult) victim originally made, not all nonconsensual porn. I imagine EU GDPR might be usable for getting nonconsensual porn removed from online platforms, but I haven't seen this tried yet.
I'm disgusted, but not surprised, that teenage kids are generating CSAM like this. Even before we had diffusion models, we had GANs and deepfakes, which were almost immediately used for generating shittons of nonconsensual porn[1].
[0] https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalit... and the later https://en.wikipedia.org/wiki/United_States_v._Handley
This is true, though "AI CSAM" is an oxymoron. There is no abuse in the creation of such works, and such it is not abuse material, unless of course real children are involved.
[0] Which, to be clear, is a good thing. Strong defamation law is a generalized censorship primitive.
I think the big difference now is that 1) it's much easier to do now, and 2) the computational requirements and (more importantly) technical skills have dramatically dropped.
We should also be explicitly aware that deep fakes are still new. GANs in 2014 were not creating high definition images. They were doing fuzzy black and white 28x28 faces, poorly, and 32x32 color images that if you squint hard enough you could see a dog (https://arxiv.org/abs/1406.2661). MNIST was a hard problem at that time and that's 10 years. It took another 4 years to get realistic faces and objects (https://arxiv.org/abs/1710.10196) (mind you, those images are not random samples), another year to get to high resolution, and another 2 to get to diffusion and another 2 before those exploded. Deep fakes were really only a thing within the last 5 years and certainly not on consumer hardware. I don't think the legal system moves much in 10 years let alone 5 or 2. I think a lot of us have not accurately encoded how quickly this whole space has changed. (image synthesis is my research area btw)
I'm not surprised that these teenagers in a small town did this. But the fact that all those adjectives exist in that order is distinct. Discussions of deep fakes like that Tom Scott video were barely a warning (5 years is not a long time). It quickly went from researchers thinking it can happen in the next decade and starting discussions to real world examples making the news in under their prediction time (I don't think anyone expected how much money and man hours would be dumped into AI).
Stability and Midjourney are also making money, but it’s largely with amateurs and people prototyping content for their own creations. A lot of single person Indy game developers are using these tools to generate assets, or at minimum first pass assets. I think a lot of media companies are producing the art for their articles or news letters etc using these tools. Whether this is enough, I don’t know.
People are genuinely way, way, way underestimating how intense the GPU shortage is.
The US breaking Google up also won't end the monopoly money. More likely one of the children will spin out with an even better margin business.
It's also weird because modern economics with tech has created a space that creates a lot of natural monopolies. Momentum is very powerful and it's the reason silicon valley companies will run at a loss for years creating a userbase. Trick is to keep them (or sell before buyer starts charging). There's what, 2 map companies and only one of them is in high usage because it's a default? Same with browsers. You can't compete because making a better product isn't enough in a system where network effects dominate the economics. Hardware companies are seeing that (along with patent issues, especially across borders). I have no idea how to think about this tbh because it is weird. In some cases breaking these companies up ultimately destroys them while in other cases, exactly what you say.
If you are doing geocoding or reverse-geocoding MapQuest outperforms Google Maps quite magnificently(YMMV). The cost is also lower and there are plans where you can keep the data.
The funny thing being that the C levels didn't give a shit about the results and went ahead with Google Maps anyway. So your point remains correct.
Most of the big tech companies we love to hate aren't natural monopolies. Google's anticompetitive moat is primarily made of inertia: they were the first to market with a halfway functional search index. Defaults are very sticky: Microsoft pushes Bing in Windows a lot because a lot of people don't know how to change the default search engine to anything else.
Browsers are monopolized because we let Apple do to iOS what we refused to let Microsoft do to Windows: make the platform browser the only option. In the heyday of Firefox, switching from IE was just a matter of downloading an installer and migrating your bookmarks and history. You can still do this (and I recommend that you do), but if you have an iPhone, it's more complicated. You can technically use an app called "Firefox", but it's using Safari under the hood. Apple won't let you port your own browser engine, which limits how you can improve the experience. So switching to Firefox on iOS is pointless.
Sometimes these monopolies interact. Google built Chrome specifically so they'd get a seat at the web standards table. They got people to use it by giving free advertising space to it on their homepage, and then they used their market position to introduce ridiculous numbers of new web APIs that every other browser vendor now has to reimplement to make Google properties work[0]. Microsoft and Opera found this to be such a hurdle that they switched their browsers over to Chromium. Firefox had to split engineering time up between implementing Chrome features, chasing down security bugs, and adopting multiprocess support, which slowed them down.
At no point is any of this 'natural'. The market can support five technologically independent browsers, but not when Google is trying to actively sabotage them.
[0] The most egregious example being Shadow DOM v0, which was never actually adopted as a standard, but used in YouTube's version of Polymer for years after Shadow DOM v1 was standardized. Firefox never implemented v0, so they were stuck running lots of polyfill code that Chrome wasn't.
That's not true. Natural monopolies also form due to network effects. You can follow the story of Bell Labs for an early example, which I suspect you're aware of. The reason for a monopoly wasn't for lack of demand it was about tragedy of the commons.
In our modern tech economics we have similar tragedy of the commons but a bit more abstracted. The thing is most products are a result of their userbase, not the product itself. Look at HN. Or look at Reddit or YouTube. It may look like circular logic (because it is a feedback loop), but the fact that everyone is publishing on YouTube makes YouTube bigger and more useful to the consumer which causes more people to publish on that platform because there are more users. It then makes it very difficult to compete because what can you do? You can make a platform that is 100x better than YouTube in respects to serving videos, search, pay to creators, and so on, but you won't win because you have no users and you won't have users without creators who aren't going to produce on your platform because there are no users. In other words, there's a first mover disadvantage. You're an early user and the site sucks because it has no content but you're a true believer. You're an early creator and your pay sucks because there are so few viewers (even if your pay per viewer is 100x your pay per video/time spent is going to be 10000x less because there are 1000000x fewer users). Look at PeerTube, Nebula, or FloatPlane. They are all doing fine but nowhere near as successful as YouTube which everyone hates on (and for good reason). Hell, when YouTube started trying to compete with Twitch they had to REALLY incentivize early creators to move with very lucrative deals because they were not buying the creator, they were buying their userbase. It should be a clear signal that there's an issue if a giant like Google has a hard time competing with Amazon.
For a highly competitive market you need a low barrier to entry so that you can disrupt. There are thousands of examples where a technology/product that is superior in every way (e.g. price and utility) but are not the market winners because network effects exist. Even things like BetaMax vs VHS is a story of network effects (I wouldn't say BetaMax dominated VHS, but not important), because what mattered was what you could get at a store or share (via your neighbor or local rental).
And I'm glad you mention Firefox, because it's a good example of stickiness. I've tried to convert many friends who groan and moan about how hard it is and make up excuses like bookmarks and literally showing them that on startup it'll export for you they just create a new excuse or say UI/UX is trash because the settings button is 3 horizontal lines instead of 3 vertical dots so they can't find it despite being in the same place or tabs are not as curved so its "unusable." You might even see these comments on HN, a place full of tech experts.
What I'm getting at here is that the efficient market hypothesis is clearly false and market participants are clearly not rational (or at least based on the conventional -- economic -- definitions)
I posted about my AI porn site pornpen.ai here last year and it reached the top of the front page. And yes, it's still going strong :D (and we've integrated SDXL and videos recently)
Edit: okay a lot of these look just bizarre as hell and seems to favour massive grotesque breast size
I wonder the damage on women this does
No code black box, and we get to tease the closed source companies for wrapping FOSS stuff.
Midjourny I'm most convinced is just a SD with a fine-tuned model. That would explain why everything looks like pixar and can't follow the prompt.
Creating illustrations for articles/presentations and stock photo alteratives are huge!
The ability to run for free on your local machine allows for far more iterations than using SaaS, and the checkpoint/finetune ecosystem the openness sprouted has created models performing way better for these use cases than standard SD.
In particular the people offering hosted models do not want to touch porn, because the first thing people do with these things is try to make nonconsensual porn of real people, which is absolutely fucking disgusting. Hence why they have several layers of filtering.
SD also has a safety filter, but it's trivially removable, and the people who make nonconsensual porn do not care about trivial things like licensing terms. My assumption is that switching to a noncommercial license would mean that Stability could later add further restrictions to the commercial use terms, i.e. "if you're licensing the model for a generator app like Draw Things, you have to package it up in such a way that removing the safety filter is difficult or impossible".
It is quite revolutionary for creative industry which is a few hundred billion in size, which is a reasonable market globally.
The thing is, I'm not entirely sure there's a paying portion of the market? Yes, I've heard of people paying for ChatGPT because it answers programming questions really well, but that's individual users, who are cost sensitive. The real money was supposed to be selling these things as worker replacement, but Hollywood unions have (rightfully) shut AI companies out of the markets where a machine that can write endless slop might have made lots of money.
OpenAI can remain in Microsoft's orbit for as long as the stupid altruist / accelerationist doomer debate doesn't tear them apart[1]. Google has at least a few more years of monopoly money before either the US breaks them up or the EU bankrupts them with fees. I don't know who the hell is pumping more money into either Anthropic or Stability.
[0] Porn. It's always porn. OpenAI doesn't want to touch it for very obvious reasons.
[1] For what it's worth, Microsoft has shown that all the AI safety guardrails can be ripped out by the money people at a moment's notice, given how quickly they were able to make the OpenAI board blink.