News for hackers. Stuff that matters.
Re:PhotoPrism: Browse Your Life in Pictures by mceachen on Tuesday July 11, 2023 @11:45AM (#36682629)
Deduplicating by SHA (so, exactly the same bytes) is straightforward, and phototprism does that: https://docs.photoprism.app/user-guide/library/duplicates/

Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics.

My first approach was to build a UID based on a series of metadata fields (like captured-at time, the shutter count of the camera body, and the geohash), but that breaks when metadata is missing or edited from their sibling variants.

Just finding what assets to compare against by time proved to be problematic in some edge cases. By storing in UTC (which is how I've built systems for the last thirty years!) made any file that didn't encode a time zones be wrong by whatever the correct offset was--almost all videos, and many RAW file formats don't have timezones. The solution I came up with was to encoding captured-at time _in the local time_, rather than UTC. I also infer or extract a milliseconds of precision for the date stamp, so files with more or less precision will still overlap during the initial captured-at query.

If you're interested, more details are here: https://photostructure.com/faq/what-do-you-mean-by-deduplica...

My main point here is to expose how deep this rabbit hole goes—I'd suggest you think about these different scenarios, and how aggressive or conservative you want to be in duplicate aggregation, and what you want to do with inexact duplicates.

Most importantly: have full offline backups before you use any deduping or asset management system. When (not if) the tool you're using behaves differently from how you expected, you won't have lost any data.

Re:PhotoPrism: Browse Your Life in Pictures by rivo on Tuesday July 11, 2023 @12:18PM (#36683167)
And if you just want to go by the pixel data, look into "perceptual hashing". https://github.com/rivo/duplo works quite well for me, even when dealing with watermarks or slight colour correction / sharpening. You could even go further and improve your success rate with Neural Hash or something similar.
Re:PhotoPrism: Browse Your Life in Pictures by mceachen on Tuesday July 11, 2023 @12:33PM (#36683414)
Unfortunately, (almost all!) image hashing don't detect color differences--they map images to greyscale first. This may be fine for many situations, but it will return the same result for a sepia tint, a full color original with incorrect white balance, and the final result you made after mucking with channels for a couple minutes.

I also found that there really isn't one "best" image hash algorithm. Using _several different_ image hash algos turns out to be only fractionally more expensive during both compute and query times, and substantially improves both precision and recall. I'm using a phash, gradient diff, and a DCT, all rendered from all three CIELAB-based layers, so they're sensitive to both brightness and color differences.

Re:PhotoPrism: Browse Your Life in Pictures by rivo on Tuesday July 11, 2023 @12:39PM (#36683512)
The library I posted uses colour information. It won't map to greyscale first.
Re:PhotoPrism: Browse Your Life in Pictures by eviks on Wednesday July 12, 2023 @01:45AM (#36690847)
is there an option to just calculate image hash (but on image data, not the full file of image data + metadata) without any transforms? So that if it matches you can be 100% certain it's the same image
Re:PhotoPrism: Browse Your Life in Pictures by j1elo on Tuesday July 11, 2023 @12:59PM (#36683831)
Deduplicating by SHA, or by exactly identical metadata, sounds so rudimentary in 2023... 10 years ago VisiPics was already doing an excellent job to help find similar or identical images in a collection, based on visual similarity!

http://www.visipics.info/

So much time later, I would hope that we expect more sophisticated and flexible tools, not less, for that task.

Re:PhotoPrism: Browse Your Life in Pictures by Bloating on Tuesday July 11, 2023 @02:13PM (#36684995)
I was trying to remember the name of this. Worked well, from what I remember
Re:PhotoPrism: Browse Your Life in Pictures by buildbot on Tuesday July 11, 2023 @02:49PM (#36685550)
I went about depdup in a similar but opposite way then you did - https://github.com/mgolub2/photodb

I used libraw to read the actual raw data from my images, ignoring possible metadata that can get changed/updated by Capture One for example. The raw data is just fed into a hashing function, to get an exact content hash. Does not work if your image is down-sampled of course, but that was actually my goal - I want to know if the raw content has changed/bit flipped, but don't care if the metadata is altered or missing.

Re:PhotoPrism: Browse Your Life in Pictures by mceachen on Tuesday July 11, 2023 @02:59PM (#36685694)
Ah! In this case, you might be happy to discover that ExifTool just added an "image content" hash--it's the SHA (or MD5 or whatever) hash of the _actual image bytes_, after ignoring the metadata header (and possible footer) payloads.

So if you, say, change the rotation of your RAW image file (which normally will just add or change the EXIF `Orientation` tag), the image content hash will stay the same.

See https://exiftool.org/ExifTool.html#ImageHashType

(of course, your current approach should prove substantively faster, but will only work with the formats that libraw supports)

Re:PhotoPrism: Browse Your Life in Pictures by dcow on Tuesday July 11, 2023 @05:50PM (#36687716)
Why is GP's approach substantially faster?
Re:PhotoPrism: Browse Your Life in Pictures by mceachen on Tuesday July 11, 2023 @06:44PM (#36688241)
His tool is in rust, so he can directly call the libraw function. ExifTool is a perl tool, so it'd be an external process that he'd need to coordinate with.
Re:PhotoPrism: Browse Your Life in Pictures by michaelteter on Tuesday July 11, 2023 @03:04PM (#36685780)
If there was ever an obvious use for AI, identifying (or at least scoring likeliness) images should be one.

Heck, we could even brute force this by resizing images via different methods and comparing checksums. But _surely_ AI can help us here.

Re:PhotoPrism: Browse Your Life in Pictures by mceachen on Tuesday July 11, 2023 @03:38PM (#36686226)
Certainly, but it depends on your goal. Models like CLIP (https://openai.com/research/clip) are designed to help describe images with an embedding, but are trained to give similar embeddings for similar image _content_, not for similar _pixels_.

Simple image hashes based on small thumbnails can do similar image detection quite well (see sibling comments for links to several different approaches).

Re:PhotoPrism: Browse Your Life in Pictures by taneq on Tuesday July 11, 2023 @04:59PM (#36687146)
> Most importantly: have full offline backups

Do this anyway. Always.

Re:PhotoPrism: Browse Your Life in Pictures by groby_b on Tuesday July 11, 2023 @06:52PM (#36688310)
If software is deduplicating images by SHA including metadata, it's missing a lot about photography workflows :) So, huge props (and I mean that - so many tools get that wrong!) on your approach of trying to identify asset groups and trying to ID members via a number of heuristics.

If you want to go deeper, I suggest grouping everything by image content - that means, at the very least, comparing images via a resolution independent hash. (E.g. average down to e.g. an 8x8 picture in greyscale for the simplest approach). (undouble does that nicely, and has a number of different approaches: https://erdogant.github.io/undouble/pages/html/index.html)

Re:PhotoPrism: Browse Your Life in Pictures by eviks on Wednesday July 12, 2023 @01:43AM (#36690836)
> actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics

Why not calculate hashes just for the image data and then do metadata conflict resolution separately?

Re:PhotoPrism: Browse Your Life in Pictures by BenTheElder on Wednesday July 12, 2023 @04:59PM (#36701596)
> Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics.

I thought takeout exported the original files? (alongside metadata entered into the application)