This precise thing is causing a funny problem in specialty areas. People are using e.g. Google Lens to identify plants, birds and insects, which sometimes returns wrong answers e.g. say it sees a picture of a Summer Tanager and calls it a Cardinal. If the people then post "Saw this Cardinal" and the model picks up that picture/post and incorporates it into its training set, it's just reinforcing the wrong identification..

That's not really a new problem, though. At one point someone got some bad training data about an old Incan town, the misidentification spread, and nowadays we train new human models to call it Macchu Picchu.

The difference between the name of an old Incan town and a modern time plant identification mistake is that maybe the plant is poisonous.

Made with gpt3

Imagine when there is an AI that is monitoring content creation and keeping tabs of original sources....

Then that's a cardinal now.

It's a cybernetic feedback system. Dalle is used to create new images, the images that people find most interesting and noteworthy get shared online, and reincorporated into the training data, but now filtered through human desire.

I wonder if human artists can demand that their work not be used for modelling. So as the robots are stuck using older styles for their creations, the humans will keep creating new styles of art.

They’ll ignore it either way.

Makes me think of Ouroboros

https://en.m.wikipedia.org/wiki/Ouroboros

Reminds me of https://en.wikipedia.org/wiki/Low-background_steel

In this situation, the low-background steel is the MS-COCO dataset, associated with the Fréchet inception distance computed by comparing the statistical divergence between the high-level vector outputs of passing MS-COCO images through Google’s InceptionV3 classifier, and passing DALL-E images (or its competitors) through it.

For now at least, there is a detectable difference in variety.

One interesting comment about this is that some models actually benefit from being fed their own output. Alphafold for instance was fed with its own 'high likelihood' outputs (as demis hassabis described in his lex friedman interview).

Training on auto generated images collected off the Internet is gonna be fine for a while since the images surfacing will be curated (ie. selected as good/interesting/valuable) still mostly by humans.

My discussion of this issue (which actually comes up in like every DALL-E 2 discussion on HN): https://www.lesswrong.com/posts/uKp6tBFStnsvrot5t/what-dall-...

The images i have created all have a watermark.. This is at least one way to filter out most images, by the same AI at least.

It’s trivial to remove the watermark and other tools put no watermark on.

This should be a step in cleaning your data to begin with. If you don't know the providence of your data then you shouldn't be even training with it.

Getting humans to refine your data is the best solution right now and many companies and researches go with this approach.

> Getting humans to refine your data is the best solution right now

Source ?

All those big models are trained with data for which the source is not known or vetted. The amount of data needed is not human-refinable.

For example for language models we train mostly on subsets of CommonCrawl + other things. CommonCrawl data is “cleaned” by filtering out known bad sources and with some heuristics such as ratio of text to other content, length of sentences etc.

The final result is a not too dirty but not clean huge pile of data that comes from millions of sources that no human as vetted and that no one in the team using the data knows about.

The same applies to large images dataset, e.g. Laon 400m that also comes from CommonCrawl and is not curated.

You can't use humans to manually refine a dataset on the scale of GPT-3 or DALL-E

Clip was trained on 400,000,000 images, GPT is roughly 180B tokens, at 1-2 tokens per word, that's 120,000,000,000 words.

At least cleaning it up is an embarrassingly parallel problem, so if you had the resources to throw incentives at millions of casual gamers, you might make a nice dent on Clip.

Alternatively, making a captcha where half the data is unlabeled, and half is labeled, forcing users to categorize data for you as they log into accounts.

But how would you know? A random string of text or an image with the watermark removed is going to be very hard to distinguish generated from human written.

s/ide/ena

I think with the terms requiring explicitly telling which images/parts were generated, they could be filtered out and prevent a feedback loop of "generated in/generated out" images. I'm sure there will be some illegal/against terms of use cases there but the majority should represent fair use.