What is the scaling like? E.g. what if it was 10 million?
How much memory would you need for ~2000 images, how slow does it get, etc.
Thx
"Doesn't sound too bad."
I doesn't? As a word of caution: If a 10k dataset takes a few minutes I would be careful how long 10MM pictures take. I predict it does not take 10MM/10k times a few minutes.
We also have an image heavy production use case that would be able to yield some nice metrics from this tool.
cdef extern int __builtin_popcountll(unsigned long long) nogil
dist = __builtin_popcountll(key ^ phash)
It would only take a couple of minutes to fill out the rest.
Can you elaborate a little bit on the performance you've observed? Can it work iteratively, basically asking one image at a time "have we already seen this?"
Would it work for text documents of varying quality aswell or is this unfeasible?
Anyone can just simply fork this kernel and add their dataset to it and perform the task!
The idea is that the activations within an image recognition network are similar for images that are similar and so you can measure the distance between two images in a space that has some semantic meaning.
It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API.
We're really excited about this library because finding image duplication is a very important task in computer vision and machine learning. For example, severe duplicates can create extreme biases in your evaluation of your ML model (check out the CIFAR-10 problem). Please try out our library, star it on Github and spread the word! We'd love to get feedback.