> Getting humans to refine your data is the best solution right now

Source ?

All those big models are trained with data for which the source is not known or vetted. The amount of data needed is not human-refinable.

For example for language models we train mostly on subsets of CommonCrawl + other things. CommonCrawl data is “cleaned” by filtering out known bad sources and with some heuristics such as ratio of text to other content, length of sentences etc.

The final result is a not too dirty but not clean huge pile of data that comes from millions of sources that no human as vetted and that no one in the team using the data knows about.

The same applies to large images dataset, e.g. Laon 400m that also comes from CommonCrawl and is not curated.

You can't use humans to manually refine a dataset on the scale of GPT-3 or DALL-E

Clip was trained on 400,000,000 images, GPT is roughly 180B tokens, at 1-2 tokens per word, that's 120,000,000,000 words.

At least cleaning it up is an embarrassingly parallel problem, so if you had the resources to throw incentives at millions of casual gamers, you might make a nice dent on Clip.

Alternatively, making a captcha where half the data is unlabeled, and half is labeled, forcing users to categorize data for you as they log into accounts.

But how would you know? A random string of text or an image with the watermark removed is going to be very hard to distinguish generated from human written.

s/ide/ena