Re:Show HN: Imagededup – Finding duplicate images made easy
by tanujjain on Monday October 07, 2019 @05:24AM
(#21178879)
Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however.
Considering the speed and memory requirements, following are the current considerations:
1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would refrain from giving exact numbers since this was done on a local system, not a very good environment for benchmarking)
2. CNN: Here, the time consuming part is encoding generation, which, in the absence of GPU would take much more time (a couple of minutes on 10k images). The retrieval part is pretty quick, but requires memory.
So, at this point, using this package on a scale of more than a couple of thousand images is not a good idea when done locally. We would however, address the scale aspect of the problem in future releases. Thanks for your question.
[ Parent
| on HN ]
Re:Show HN: Imagededup – Finding duplicate images made easy
by flud on Monday October 07, 2019 @05:32AM
(#21178920)
Doesn't sound too bad. But can you elaborate on the last part, about retrieval requiring memory and not scaling to more than a couple of thousands images?
How much memory would you need for ~2000 images, how slow does it get, etc.
Thx
Re:Show HN: Imagededup – Finding duplicate images made easy
by tanujjain on Monday October 07, 2019 @05:57AM
(#21179081)
Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n' increases. We have already made some optimizations to reduce the memory footprint but there would be clear upper limits to it (We haven't experimented). As for the numbers, the cifar10 dataset example present in the repo was carried out on google colab notebook. The dataset has 60k images and ended up consuming about 6G of RAM using CNN. However, since there hasn't yet been a principled benchmarking done on the package, I would consider these numbers as only marginally indicative of the capabilities. A better way to figure out if it works for you would be to try it out with your own dataset, starting out at a small scale and increasing the dataset size gradually.
Re:Show HN: Imagededup – Finding duplicate images made easy
by dgacmu on Monday October 07, 2019 @07:04AM
(#21179438)
Use approximate nearest neighbor. The FAISS library is good.
Re:Show HN: Imagededup – Finding duplicate images made easy
by tanujjain on Monday October 07, 2019 @08:30AM
(#21179975)
Thanks for the pointer, will check it out.
Re:Show HN: Imagededup – Finding duplicate images made easy
by jonatron on Monday October 07, 2019 @07:45AM
(#21179668)
Couldn't you add images>threshold to a dict/map as you iterate, rather than building a complete matrix, then iterating through that?
Re:Show HN: Imagededup – Finding duplicate images made easy
by tanujjain on Monday October 07, 2019 @08:24AM
(#21179933)
We tried that approach, but it was way too slow.
Re:Show HN: Imagededup – Finding duplicate images made easy
by jayalpha on Monday October 07, 2019 @06:39AM
(#21179288)
"The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes"
"Doesn't sound too bad."
I doesn't? As a word of caution: If a 10k dataset takes a few minutes I would be careful how long 10MM pictures take. I predict it does not take 10MM/10k times a few minutes.
Re:Show HN: Imagededup – Finding duplicate images made easy
by gravypod on Monday October 07, 2019 @11:16AM
(#21181827)
What way is the hash represented? Have you looked at NN libraries like faiss [0] and NGT [1]? Those can quite easily handle a nearest neighbor search of 10 million vectors and, from my understanding, they turn their vectors into some kind of hash that is then searched.
Re:Show HN: Imagededup – Finding duplicate images made easy
by tanujjain on Tuesday October 08, 2019 @03:28PM
(#21196161)
The hashes are 16 character hexadecimals represented as strings. Had a quick look at the faiss package and it looks promising. Would consider it for the next versions.
Re:Show HN: Imagededup – Finding duplicate images made easy
by gravypod on Tuesday October 08, 2019 @05:25PM
(#21197437)
If you're interested in collaboration I'd be happy to help with a prod-focused version. My work has a need for a shardable daemon for dedup tasks. My personal email is in my description and I'm also available via [email protected].
We also have an image heavy production use case that would be able to yield some nice metrics from this tool.