News for hackers. Stuff that matters.
Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @04:39PM (#25063339)
> indexes them in a local SQLite DB.

I don't really understand this. I've been in computing for 45 years now. Programs come and go. I store files I want to keep in the most generic format practical. Storing a file in some program's special format is not a good plan for reading it 40 years from now.

For photos, I store them as jpgs in folders named after the year. Within those folders, there might be sub-folders named with a topic, like "disneyland" or "christmas". I'll "tag" photos by selecting a name for the jpg, like "bob and sue.jpg".

If more is needed, I'll just add a "notes.txt" file in the folder, with whatever text seems appropriate.

I have no worries about ascii text becoming unreadable, and few worries about jpgs becoming unreadable.

Re:Google Photos will end its free unlimited storage in June... by mholt on Wednesday November 11, 2020 @04:57PM (#25063519)
Please don't misunderstand, with your 45 years of computing experience. :) (which I highly respect) I never said the files are stored in sqlite. They are just indexed in sqlite. The files are stored on disk like normal.
Re:Google Photos will end its free unlimited storage in June... by peterwwillis on Wednesday November 11, 2020 @09:38PM (#25065808)
> Please don't misunderstand

If there's one thing people will always do regardless of age, it's that.

Re:Google Photos will end its free unlimited storage in June... by phoe-krk on Wednesday November 11, 2020 @04:46PM (#25063411)
SQLite is an ubiquitous and well-tested piece of software that is going to stand the test of time and survive the next twenty years. There are very few pieces of software that I am able to praise like that with a clean conscience; SQLite is definitely one of them though. You might find it worthwhile to read e.g. about its memory allocation strategy, or its testing strategy.

Also, I don't think that the image files themselves are stored in the SQLite DB file; likely just their indices and metadata.

Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @05:06PM (#25063586)
> SQLite is an ubiquitous and well-tested piece of software that is going to stand the test of time and survive the next twenty years.

My timeline is much longer than 20 years. I have family photos going back to the Civil War.

Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.

I switched my mail from Outlook to Thunderbird because the former stores the email in some undocumented unreadable binary format. TB stores the mail as text, so I can recover the mail without needing TB. I have mail going back 25 years now. Some of my earlier mail is now lost because, surprise surprise, the mail program no longer works and the data is stored in a proprietary format.

I unzipped all my old file archives a few years back out of concern that some of the old DOS archive software would disappear.

Re:Google Photos will end its free unlimited storage in June... by williadc on Wednesday November 11, 2020 @05:59PM (#25064037)
> Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.

Here's a Hacker News story from 10 days ago about an update to Wordperfect: https://news.ycombinator.com/item?id=24959090

I haven't tried it myself.

Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @07:13PM (#25064627)
I saw that story, which is why I remembered Wordperfect. Good luck having a DOS emulator that works with WP 40 years from now :-)

Now what happens if it was a slightly less popular word processor, like PC-Write? (PC-Write rose to fame by being the first shareware program.)

Re:Google Photos will end its free unlimited storage in June... by fomine3 on Wednesday November 11, 2020 @07:52PM (#25064987)
I agree to store metadata inside the file is better, but FYI: SQLite is expected to be supported very long.

https://www.sqlite.org/lts.html

Re:Google Photos will end its free unlimited storage in June... by knyazhefilms on Wednesday November 11, 2020 @11:02PM (#25066299)
>My timeline is much longer than 20 years.

What storage format do you use then? Are you concerned about Hard drives format becoming obsolete? Are you using Mac or Windows? What if tomorow Apple or Microsoft decide to stop support for their formats? What would you consider as a reliable Storage format? Do you trust linux formats ext3 ext4? What would be your ideal choice looking from your perspective which I really share. It's really a big question for me how reliably store things for long periods of time.

Re:Google Photos will end its free unlimited storage in June... by WalterBright on Thursday November 12, 2020 @06:42AM (#25068777)
I copy them forward to new media every year or so. It's how I avoid bit rot and media obsolescence.

My files have survived magtapes, DECtapes, 8" floppies, 5.25" floppies, 3.5" floppies, zip drives, CDs, DVDs, blurays, 6 Mb hard disks (yes, 6 megabytes!), many no-longer-readable hard drives, and that about covers it.

Just keep copying them forward, and rotate among more than one drive.

My experience is that every media device and every media format becomes unreadable after a few years.

I learned programming on punch cards. I wish I'd kept my punch card decks, but they'd be unreadable by now. Unreadable by machine, that is, I could decode them by hand.

Re:Google Photos will end its free unlimited storage in June... by stjohnswarts on Wednesday November 11, 2020 @05:57PM (#25064026)
I hope you at least have at least 1 on site and one off site and 1 physical media backup of all those if you want to keep them for the next few generations...
Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @07:14PM (#25064635)
Having all your backups in the same spot is as dumb as relying on the cloud.
Re:Google Photos will end its free unlimited storage in June... by depr on Wednesday November 11, 2020 @05:48PM (#25063935)
Wordperfect files can be opened in MSFT Word ;)
Re:Google Photos will end its free unlimited storage in June... by google234123 on Wednesday November 11, 2020 @05:57PM (#25064027)
For how much longer? And how much resources do you imagine will be spend to fix bugs/issues opening these files? I don't think this example is strong enough to deserve a mocking wink
Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @04:57PM (#25063513)
> likely just their indices and metadata

notes.txt will work just fine, then, even for several thousand photos. I don't see a need for a database until you've got far more than that. The tree file system works tolerably well as a "database".

As for searching, I know how to use "grep" and "locate".

Re:Google Photos will end its free unlimited storage in June... by joshuamorton on Wednesday November 11, 2020 @05:36PM (#25063846)
Tags are more flexible than folders. An actual index let's you search by tag. Folders don't let you search for all photos of your mother, or in 1987, or in Alabama, or some combination thereof.

Folders are a limiting api and we shouldn't limit ourselves to strictly hierarchical organizational structures for non hierarchal data.

Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @05:59PM (#25064040)
I know, but folders work perfectly fine even up to a few thousand files. It doesn't take me long to find a vacation picture from 1966, for example. I look in the 1966 folder, which doesn't have a lot in it. The preview thumbnails from file explorer quickly let me hone in.

It'd take me far more time to set up a proper tag database than I'd save looking things up.

It's still infinitely better than a random shoebox with random snapshots in it.

Re:Google Photos will end its free unlimited storage in June... by sokoloff on Wednesday November 11, 2020 @08:30PM (#25065339)
Looking up something by its clustered index is indeed pretty easy. That's not the only access pattern into most data sets, which is where tags can be handy.

I agree with your last two points, as my family's digital photos are also stored in a fairly similar clustered index style where you have to know the date at least pretty closely to start your search unless you want to scan "all Thanksgivings" [which itself is just a series of clustered index lookups]

Re:Google Photos will end its free unlimited storage in June... by megablast on Wednesday November 11, 2020 @06:41PM (#25064362)
45 years experience, and your first thought is that they are being stored as a binary blob in a database??
Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @07:25PM (#25064732)
Yes. Binary file formats are common (like Microsoft Outlook, pdf files, etc.), and are very fast to read/write. Text files have to be parsed. I've written lots and lots of dumpers for binary file formats generated by other programs.
Re:Google Photos will end its free unlimited storage in June... by mholt on Wednesday November 11, 2020 @11:13PM (#25066362)
Oof, I feel the trauma on this one. I've been bitten by too many binary file formats that I don't care to store aggregate amounts of content in them (unless it's just metadata).
Re:Google Photos will end its free unlimited storage in June... by ForHackernews on Wednesday November 11, 2020 @04:44PM (#25063395)
TBF SQLite format is also extremely unlikely to become unreadable: https://www.sqlite.org/fileformat2.html
Re:Google Photos will end its free unlimited storage in June... by Retric on Wednesday November 11, 2020 @04:44PM (#25063394)
Metadata is more useful with older photos. Right now it’s obvious who’s in them, but in 40 years being able to just put names to faces can be difficult.
Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @04:49PM (#25063449)
I just note their names in the file name. If there are too many, I make a copy of the photo, bring it up in an image editor, and type the names of the people next to their heads. It works like a champ.
Re:Google Photos will end its free unlimited storage in June... by zeroxfe on Wednesday November 11, 2020 @06:29PM (#25064276)
Bits rot. If you want your data to last the next 40 years, plan for spending a few hours every 5 years migrating your storage. It's a far better plan than trying to predict what the world will be like 40 years from now.
Re:Google Photos will end its free unlimited storage in June... by WalterBright on Wednesday November 11, 2020 @07:19PM (#25064683)
Absolutely. I buy new drives every year or so and copy the stuff forward. Many of my 10+ year old drives no longer function.
Re:Google Photos will end its free unlimited storage in June... by e_y_ on Wednesday November 11, 2020 @05:26PM (#25063753)
It would be easy enough to use the SQLite DB for normal read/write and search, and also save a human readable .txt dump periodically such as when making backups, or asynchronously on every save.
Re:Google Photos will end its free unlimited storage in June... by kristopolous on Wednesday November 11, 2020 @08:17PM (#25065210)
Creating a cadence that also stores the data in EXIF isn't really that hard if that's your concern. You could even get SQLite triggers to do external calls to update the tags if things change. This is all pretty trivial.