It was hard to accomplish, but you can finetune SD on your computer. They are working on instruction-tuning LLMs as well. In general ML models are not closed boxes inaccessible to us - they can be finetuned, reprompted, you can even average two versions to get a mix of two models. In the last 2 years lots of papers were written on finetuning and prompting, all of them geared towards low resource AI adaptation to new tasks.
- lexical filtering by applying a blacklist of artist names on the original prompt
- perceptual filtering - drop all generated images that look too close to copyrighted images in your training set
- re-captioning based filtering - use a model to generate captions for an image and apply filters on the captions; you can also filter by visual style
- CLIP based filtering where you use embeddings to find nearest neighbours, and if they are copyrighted then you can drop the image
- or train a copyright violation detection model that takes generated images and compares them to images from the original authors
Copyright enforcement struggles are going to be interesting to watch in this decade. But I think it will slowly become irrelevant, because anything can be generate again slightly different until they finally pass the filters.
This is part of the motivation for “foundation models”.
There’s another paradigm called student/teacher models where a randomly initialized model updates it’s weights according to another pretrained model. This could (maybe?) be used to achieve the desired effect of a model that learned in a “clean room”.
Because it's already good enough to have made it's way into many of my workflows.
I do feel that many companies will, ironically, use "ethical" as a pretext to not be open.
I mean this isn't even speculative anymore after what happened with - hilariously named - OpenAI
> that many companies will, ironically, use "ethical" as a pretext to not be open.
Yes, weaponized ethics as sleight of hand for control is a common historical pattern.
This is exactly why I pay $30 per month for MidJourney. The output is just phenomenally better than most of the images coming out of SD, and the UI is much better as well. It's just not worth my time fiddling with SD if the results are so bad in comparison.
If/when SD catches up, I'd jump ship to using it in a heartbeat.
While it'd be difficult to improve upon the model, it might be easy enough to finetune it if needed, and it's certainly worth it to USE it as is.
There is a limited number of models costing 6 digits in dollars in train time and are freely available. There is certainly value in preserving them, in a world of artificial scarcity.
Is it possible to crowdsource AI training with something that looks similar to folding@home?
AI thrives and depends on large amounts of clean, well labeled data.
Large corporations understand this and have hoarded data for a long time now. Some of them have also managed to label this data by millions of people through things like Recaptcha, or just by hiring lots of people to do it.
Open datasets tend to be much smaller and dirtier than small, open projects have access to.
I suppose it would be possible to, over time, collect lots of data and crowd-source some project to clean it up and label it well enough to be useful, then crowd-source the AI model training itself, but it would probably take a long time and by then corporate-owned AI models will already dominate (as they do now with MidJourney, for example, being way better in my experience than Stable Diffusion, but with time the difference will only get starker).
I'd also be concerned with such ostensibly open projects eventually going closed and commercial as IMDB did after getting lots of work by volunteers freely giving their time to writing reviews.
More recently the open LAION data sets have become widely used by both tech giants and independent researchers.
The problem is DL is really sensitive to dirty data, disproportionately so.
At $DAYJOB once we cleaned the dataset, removed a few mislabeled identity/face pairs (very few, about 1 in 1e4) and the metrics goes up a lot.
In fact, generally DL is quite tolerant to label noise, especially using modern training methods such as SSL pretraining.
A fair compromise is any marketplace for clusters with good interconnect but a lot cheaper than the cloud. Tuning distributed training and network transport layer for settings not as homogeneous as the cloud will also help on top of generally good interconnect. Security is a concern.
Building on points raised by pmoriarty, being able to scrape data makes up for lacking labeled data in the era of self-supervised training. IP-hawks are now putting a damper on that option, which is why I worry this might backfire from a freedom perspective.
I even think there will be multiple initiatives like this, and there will be at least 1 big repository that accepts inputs and retrains periodically for anyone who wants the model.
The quality of the output you can get with the models right now have perpetual utility IMO. If you use it to create patterns, backgrounds, or even just for inspiration creations right now, it might be a shame if it didn't progress (depending on your position) but it's fine as-is if you put in the work to compose and refine the raw output.
I wonder if that will continue.
My understanding is that's partially because it currently relies on GPUs, which until relatively recently there was a limited demand for, and the market is basically controlled by a single company.
Will we see cheaper special purpose AI accelerators? Like happened with crypto mining ASICs.
LLMs have the unfortunate limitation of being both powerful and lending themselves to centralized control choke-points due to how resource intensive they are to train. Under this paradigm, I fear commercial entities will be able to easily navigate the legal landmines and continually improve while open efforts perpetually lag far behind.
There are many vested interests who want this control for various reasons they justify as: protection from x-risk, keeping it out of the hands of abusers and bullies, economic advantage. Their reasons for want of control are either well intended but wrong-headed or profit-motivated and disingenuous.
Rather than challenging the likes of GPT-3 and Copilot enabling freedom, I fear folks will be forced to send all their videos, pictures, text and code to the servers of Microsoft, Amazon and Google or lose access to advantages as LLMs continue to improve at a rapid clip.