News for hackers. Stuff that matters.
Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @02:58PM (#25062275)
I have one primary collection but I also have probably 30TB of old drives lying around with photos scattered all about. Is there something I can do to mount those drives, grab all image files and then dedupe/catalog them (at least by date)? Ideally this would work with videos as well.
Re:Google Photos will end its free unlimited storage in June... by bayindirh on Wednesday November 11, 2020 @04:12PM (#25063041)
Considering you're using Linux, I'd do something of sorts:

  1. get all image files with "find", considering they are with a known extension
  2. run jdupes on the dump and deduplicate them.
  3. run exiftool on them to automatically divide them to folders based on any metadata field you like.
  4. Index all of the images with digikam and further organize them there.
Another path would be to add all drives as "removable collections" to digikam and manage all of them there. digikam also has fuzzy search so it can find not only identical but similar images so you can deduplicate them.

Both ways are applicable to videos as well.

I'm currently using the second path since Digikam is already my primary photo cataloging and managing tool for years and, it works wonders.

---

On mac, Gemini II and Retrobatch would allow for a similar workflow but, I didn't use them as my primary workflow tools. Gemini also has similarity search so it can deduplicate similar photos.

I'm not using Windows for more than a decade so, I don't know anything on that front.

Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @10:17PM (#25066016)
This looks like a great approach, thanks for putting it together. I'm hoping to dump these disks to an image b/c the filesystems on them are all over the place...some of them are IDE lol, then just loopback mount them for this process.
Re:Google Photos will end its free unlimited storage in June... by bayindirh on Thursday November 12, 2020 @01:35AM (#25067094)
I'm glad that it helped. Organizing old files always take much longer than anticipated. It's a live exercise on entropy :)

Taking images and mounting them sounds reasonable. :)

Re:Google Photos will end its free unlimited storage in June... by Jiocus on Wednesday November 11, 2020 @04:20PM (#25063117)
From the comfort of a terminal[1], a few options comes to mind,

git-annex[2] will allow you to index all, or just some, of those files where they are - and keep track if you shuffle them around. The really useful feature in your case, is that git-annex will keep tabs on even your disconnected harddrives, flashdrives or cloud storage. It will let you know if you have redundant copies and how many, or if you're about to trash the last known instance of IMG001.jpg. It will point you to specific storage media if query some file not currently local.

Note that it's not entirely as trivial as I make it out to be - git vcs experience helps. Some love it.

In your situation, I'd might try borg[3] - No experience, but I heard appreciative voices about it and docs seem OK.

Personally, I always end up using rmlint/fdupe and unix tools, but that's a secret.

[2] https://git-annex.branchable.com/ [2] https://github.com/borgbackup/borg [3] There's GUI implementations of these

Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @10:15PM (#25066007)
Whoa git-annex looks like it would be useful for a few projects, thanks for the tip. Will take a look at both.
Re:Google Photos will end its free unlimited storage in June... by jdeibele on Wednesday November 11, 2020 @04:03PM (#25062947)
I've been pleased with PhotoSweeper on my Mac. I started with the free Lite version, then went ahead and bought the full version.

I set it at the highest match setting. If you adjust it to a lower setting, it will match things where somebody is looking at the camera versus looking away.

https://apps.apple.com/us/app/photosweeper/id463362050?mt=12 https://apps.apple.com/us/app/photosweeper-lite/id506150103?...

PS: I was going to say that they have a PC version but it looks like the 3rd entry for me on DuckDuckGo is actually spam that says there is a PC version, then feeds you to programs that are "like PhotoSweeper". I wouldn't download those.

Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @10:14PM (#25066001)
Will check this out, thanks for the suggestion!
Re:Google Photos will end its free unlimited storage in June... by avgDev on Wednesday November 11, 2020 @03:07PM (#25062380)
I bought a Synology NAS mainly for images/videos/personal files and run hard drives in RAID setup. You could explore other NAS solutions like QNAP or open source.
Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @03:36PM (#25062685)
I've stopped and started shopping for a NAS at least five times now. One of these days i'll pull the trigger.
Re:Google Photos will end its free unlimited storage in June... by alliao on Wednesday November 11, 2020 @05:32PM (#25063803)
Synology's photo station is slightly jarring... the whole indexing operation isn't optimized at all so if you have like 500k photos it's going to be spinning for a while.
Re:Google Photos will end its free unlimited storage in June... by walterbell on Wednesday November 11, 2020 @03:12PM (#25062444)
There's an ancient-but-maintained image cataloger / asset manager called NeoFinder, available on Windows and Mac. Would love to find an OSS equivalent.

https://cdfinder.de/

Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @03:35PM (#25062678)
Will take a look, thank you!
Re:Google Photos will end its free unlimited storage in June... by x87678r on Wednesday November 11, 2020 @03:13PM (#25062450)
Same problem. The other issue is having a 2 cameras and 2 phones in the family and grouping together events. Its easier just to save them all and not try.
Re:Google Photos will end its free unlimited storage in June... by jcims on Wednesday November 11, 2020 @03:27PM (#25062605)
My problem is I'm paranoid of getting rid of anything now. I just want to take one good sweep through so I can do something with those drives.