I don't really understand this. I've been in computing for 45 years now. Programs come and go. I store files I want to keep in the most generic format practical. Storing a file in some program's special format is not a good plan for reading it 40 years from now.
For photos, I store them as jpgs in folders named after the year. Within those folders, there might be sub-folders named with a topic, like "disneyland" or "christmas". I'll "tag" photos by selecting a name for the jpg, like "bob and sue.jpg".
If more is needed, I'll just add a "notes.txt" file in the folder, with whatever text seems appropriate.
I have no worries about ascii text becoming unreadable, and few worries about jpgs becoming unreadable.
If there's one thing people will always do regardless of age, it's that.
Also, I don't think that the image files themselves are stored in the SQLite DB file; likely just their indices and metadata.
My timeline is much longer than 20 years. I have family photos going back to the Civil War.
Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.
I switched my mail from Outlook to Thunderbird because the former stores the email in some undocumented unreadable binary format. TB stores the mail as text, so I can recover the mail without needing TB. I have mail going back 25 years now. Some of my earlier mail is now lost because, surprise surprise, the mail program no longer works and the data is stored in a proprietary format.
I unzipped all my old file archives a few years back out of concern that some of the old DOS archive software would disappear.
Here's a Hacker News story from 10 days ago about an update to Wordperfect: https://news.ycombinator.com/item?id=24959090
I haven't tried it myself.
Now what happens if it was a slightly less popular word processor, like PC-Write? (PC-Write rose to fame by being the first shareware program.)
What storage format do you use then? Are you concerned about Hard drives format becoming obsolete? Are you using Mac or Windows? What if tomorow Apple or Microsoft decide to stop support for their formats? What would you consider as a reliable Storage format? Do you trust linux formats ext3 ext4? What would be your ideal choice looking from your perspective which I really share. It's really a big question for me how reliably store things for long periods of time.
My files have survived magtapes, DECtapes, 8" floppies, 5.25" floppies, 3.5" floppies, zip drives, CDs, DVDs, blurays, 6 Mb hard disks (yes, 6 megabytes!), many no-longer-readable hard drives, and that about covers it.
Just keep copying them forward, and rotate among more than one drive.
My experience is that every media device and every media format becomes unreadable after a few years.
I learned programming on punch cards. I wish I'd kept my punch card decks, but they'd be unreadable by now. Unreadable by machine, that is, I could decode them by hand.
notes.txt will work just fine, then, even for several thousand photos. I don't see a need for a database until you've got far more than that. The tree file system works tolerably well as a "database".
As for searching, I know how to use "grep" and "locate".
Folders are a limiting api and we shouldn't limit ourselves to strictly hierarchical organizational structures for non hierarchal data.
It'd take me far more time to set up a proper tag database than I'd save looking things up.
It's still infinitely better than a random shoebox with random snapshots in it.
I agree with your last two points, as my family's digital photos are also stored in a fairly similar clustered index style where you have to know the date at least pretty closely to start your search unless you want to scan "all Thanksgivings" [which itself is just a series of clustered index lookups]
There are a lot of converging ideas out there, for "self-hosted NSA of yourself" type software which gather your own data from services you use and index it / make it more accessible to users. Would love to see something serious come out of it.
I followed Timeliner early on and really like the idea, but yeah it's a bitch to get traction for this sort of thing. You need people willing to implement backends for basically everything, then maintain them.
I mean, it's a small amount, but on the scale of google it might be worth it.
Each photo's geolocation data is stored. Of course.
It is shown in the UI, as well as indexed so you can search by location.
For example, I can search for "photos of dogs in Portland" and it'll show all the pics of dogs I've taken in Portland.
1. get all image files with "find", considering they are with a known extension
2. run jdupes on the dump and deduplicate them.
3. run exiftool on them to automatically divide them to folders based on any metadata field you like.
4. Index all of the images with digikam and further organize them there.
Another path would be to add all drives as "removable collections" to digikam and manage all of them there. digikam also has fuzzy search so it can find not only identical but similar images so you can deduplicate them.Both ways are applicable to videos as well.
I'm currently using the second path since Digikam is already my primary photo cataloging and managing tool for years and, it works wonders.
---
On mac, Gemini II and Retrobatch would allow for a similar workflow but, I didn't use them as my primary workflow tools. Gemini also has similarity search so it can deduplicate similar photos.
I'm not using Windows for more than a decade so, I don't know anything on that front.
Taking images and mounting them sounds reasonable. :)
git-annex[2] will allow you to index all, or just some, of those files where they are - and keep track if you shuffle them around. The really useful feature in your case, is that git-annex will keep tabs on even your disconnected harddrives, flashdrives or cloud storage. It will let you know if you have redundant copies and how many, or if you're about to trash the last known instance of IMG001.jpg. It will point you to specific storage media if query some file not currently local.
Note that it's not entirely as trivial as I make it out to be - git vcs experience helps. Some love it.
In your situation, I'd might try borg[3] - No experience, but I heard appreciative voices about it and docs seem OK.
Personally, I always end up using rmlint/fdupe and unix tools, but that's a secret.
[2] https://git-annex.branchable.com/ [2] https://github.com/borgbackup/borg [3] There's GUI implementations of these
I set it at the highest match setting. If you adjust it to a lower setting, it will match things where somebody is looking at the camera versus looking away.
https://apps.apple.com/us/app/photosweeper/id463362050?mt=12 https://apps.apple.com/us/app/photosweeper-lite/id506150103?...
PS: I was going to say that they have a PC version but it looks like the 3rd entry for me on DuckDuckGo is actually spam that says there is a PC version, then feeds you to programs that are "like PhotoSweeper". I wouldn't download those.
https://exiftool.org/forum/index.php?topic=10412.msg55391#ms...
The project seems like it could be very interesting, but there's too much effort required to perform an initial evaluation.
Contributions welcomed to lower the barrier to entry. Use it, contribute to it, or leave it, I guess!
But yeah, not enough for me to pay them for it. If I'm going to pay someone, I'll pay someone else.
Yandex Disc also has an AI which allows you to search for objects within your photos.
I had to switch to using a third party app to sync my "photo reel" from my phone to Google Drive: https://play.google.com/store/apps/details?id=com.ttxapps.dr...
Then I use Google Backup & Sync to create a local copy on my desktop, scheduled SyncToy job to copy it to the NAS, and the NAS has a scheduled job to backup to Backblaze B2. In my setup, these irreplaceable photos & videos exist on my phone, Google, my desktop, my NAS, and Backblaze.
If you are relying on Google Backup & Sync to restore a large amount of data after a data loss event, you are in for a world of pain and disappointment. It simply does not handle any significant amount of data well. I had to stop backing up "My Computer" with Backup & Sync and just switched to using Google Drive. The files in Google Drive seem to be the only way to "restore" a large amount of files after a data loss event. If they are backed up under "My Computer" in Google Drive and you have a decent amount of data (50GB+), you will have no way to restore these files. The web interface will simply time out when you try to download them. The Backup & Sync client won't even attempt to download these files, even if you try the dead-end workarounds suggested on the web. You won't be able to drag them into your "Google Drive" folder as a workaround either. The whole thing is atrocious.
Google Backup & Sync is a fucking joke, and Google should be ashamed of themselves for releasing such a shitty product which does not actually help consumers protect against data loss. Consumers will only realize this at the point they are fucked and trying to restore their data.
Onedrive is of course a different product than Google Photos, and the last time I used Google Photos, Google Now was still a left-swipe sidebar, so YMMV.
I wanted to look at pictures on the big screen of my desktop Mac. Leaving Google Photos in automatic upload on my phone meant that photos would all get uploaded but the ones that I culled would be deleted from Apple Photos but not Google Photos.
Using Google Photos on my phone would let me delete both copies (Apple and Google) but it was much harder to make out differences. 27" screen vs 6" screen.
I was using Google Photos in the unlimited free quality mode so their copy shouldn't match my original pictures.
So Google Photos ended up as a place where I dumped photos as a backup. I bought more storage on iCloud.
So, just do your deletion on the Google Photos web client on desktop. The culled photos will be culled from Apple Photos as well.
Given how they often they deprecate things and how hard I've heard it is to restore a disabled account, I feel like some kind of smaller company dedicated to photo storage would be much better anyway
I would be curious to know what the competition looks like, but I don't know of any other service that lets me do "Show me photos of me and my brother at the beach" and actually return relevant results. The face/object clustering is near flawless, and even works with pets.
And as for killed services with a billion users, well, free unlimited Google Photos is now a killed service.
As for killed paid services, Google Play Music, Nest Secure, Google Photos Print, Google Audio Ads, and a ton of others.
Not really, it's mostly frequency illusion hanging out on HN, but considering Google has multi-billion users, those dozen or so incidents are nothing in comparison.
> free unlimited Google Photos
That's a feature, not a service. And name me one other company that provides unlimited free storage.
> Google Play Music
Technically migrated, all your data and purchases are still there
> Nest Secure
Hardware is a completely different ball game. The existing users can still keep using their hardware just fine. Every single device eventually stops being sold.
> Google Photos Print
Again, a feature. You didn't lose any data. You can still print photos using other services.
> Google Audio Ads
Not familiar, but I see it's still around? Do you have a link to the announcement of it being killed?
Given the scale of Google and how many people have access to the Internet today, I'd generally be willing to bet that there are many, many others with the same fate but who simply can't get enough traction behind them for anyone (let alone Google) to care enough.
It's not at all uncommon.
I've done business with Google multiple times in my career, and I could name a dozen such incidents. They're just not public.
For the most part, there wasn't anything the user did; just Google algorithm bugs. In one case, a startup lost all of its data because one Google system expected another Google system to implement a fraud detection measure, and the other Google system didn't. Completely internal to Google, but poof, all of a sudden, account gone, all data gone, and no way to fix it.
In another case, GCE lost a contract worth millions of dollars to AWS because Youtube had algorithm bugs. Poof. Youtube system broke. This was early GCE days, and they were looking for successes, and we were sufficiently high-profile that we had a dedicated engineering team. They pushed on Youtube. Youtube said we weren't important enough to help.
Most of this stuff is either under NDA, or otherwise just private, but it is why my current employer, as a policy, doesn't do business with Google. That's a big contract too; I'm not at a startup right now, but at a big company.
> And name me one other company that provides unlimited free storage
I think you're unintentionally arguing I shouldn't rely on Google-unique features, since I might lose them. I already knew that. I've learned that lesson painfully over, and over, and over, and over, and over.
That hurts Google in B2B. If all customers take that philosophy, Google fundamentally can't have unique, differentiating advantages beyond price. It doesn't matter what unique AI algorithm Googlers come up with for the Google Cloud. I won't use it if I can't count on it still being around when I ship my product. And I can't. The only time I will use Google is if it provides unique customer access (e.g. Android apps or Youtube eyeballs), which hopefully the antitrust thing will help with.
I still like Google Photos, but ideally I'd automatically sync all my photos to a backup service and have things set up so I could make it the primary replica if needed.
I can easily fill up that much data in a single month with videos of my kids.
The next tier is 2TB for $10/mo. Then 10TB for $50/mo.
All of a sudden, you're paying $600 a year and it isn't as insignificant as you suggested.
Then after backing them up locally, I won't feel bad about deleting them from the cloud later to free up space.
I've been using Timeliner for a while but need to update it. New maintainers welcomed, if you're interested!
(One major "oof" is that the Google Photos API strips geolocation data, so unfortunately coordinates are lost when using this method. There's discussion about using Takeout as a workaround, or even automating web browser interactions, but those have their own problems too.)
[1]: https://github.com/mholt/timeliner