Quantifying Memorization Across Neural Language Models https://arxiv.org/abs/2202.07646
Deduplicating Training Data Makes Language Models Better https://arxiv.org/abs/2107.06499 https://twitter.com/arankomatsuzaki/status/14154721921003397... https://twitter.com/katherine1ee/status/1415496898241339400
When my prompt isn't working, I often want to check whether the concepts I use are even present in the dataset.
For example, inputting `Jony Ive` returns pictures of Jony Ive in Datasette and pictures of apples and dolls in clip retrieval.
(I know laion 5B is not the same as laion aesthetic 6+, but that's a lesser issue.)
[0] - https://rom1504.github.io/clip-retrieval/
[1] - https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...
It works for your example
I guess I'll disable it by default since it seems to confuse people
Using clip for searching is better than direct text indexing for a variety of reasons but here for example because it matches better what stable diffusion sees
Still interesting to have a different view over the dataset!
If you want to scale this out, you could use elastic search
---
Also: There is a joke to be made at Jony's expense regarding the need to turn off aesthetic scoring to see his face.
Nice tool!
You can also explore the dataset there https://rom1504.github.io/clip-retrieval/
Thanks to approximate knn, it's possible to query and explore that 5B datasets with only 2TB of local storage, anyone can download the knn index and metadata to run that locally too.
Regarding duplicates, indeed it's an interesting topic!
Laion5b deduplicated samples by url+text, but not by image.
To deduplicate by image you need to have an efficient way to compute whether image a and b are the same.
An idea to do that is to compute an hash based on clip embeddings. A further idea would be to train a network actually good at dedup and not only similarity by training on positive and negative pairs, eg with triple loss.
Here's my plan on the topic https://docs.google.com/document/d/1AryWpV0dD_r9x82I_quUzBuR...
If anyone is interested to participate, I'd be happy to guide them to do that. This is an open effort, just join laion discord server and let's talk.