News for hackers. Stuff that matters.
Re:Files Are Hard (2015) by cjfd on Sunday December 01, 2019 @10:33AM (#21675910)
It has been a while ago so maybe things are better now but I have seen sqlite being a disaster when the times comes to upgrade the schema. Not all of the common operations like deleting/renaming a column are supported. Since it has been a while I don't remember exactly which ones. Then someone writes an sql script to provide this functionality. Then it turns out that a freshly created database using the latest schema is subtly different from one that was created from an older schema and then updated. Things like a column that has a default but only in one of these two cases. Lots of fun, but not really.
Re:Files Are Hard (2015) by wruza on Sunday December 01, 2019 @10:40AM (#21675939)
I think what gp meant is fopen-way of sqlite, not full-schema. Basically, tXML/tJSON singleton and tBlob(id, data) for embedded blobs. That way it will have all journaling, syncing, checksums and other fs wisdom that the article mentioned.
Re:Files Are Hard (2015) by blattimwind on Sunday December 01, 2019 @10:37AM (#21675925)
For most uses of SQLite it's acceptable (or even advantageous) to copy the entire database for doing an upgrade, instead of the more RDBMS-y way of using DDL.
Re:Files Are Hard (2015) by Hackbraten on Sunday December 01, 2019 @11:20AM (#21676136)
Which brings you the additional challenge of how to overwrite the old database file with the new one atomically in the face of crashes.
Re:Files Are Hard (2015) by tylerhou on Sunday December 01, 2019 @11:50AM (#21676301)
http://man7.org/linux/man-pages/man2/rename.2.html

> If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.

Re:Files Are Hard (2015) by Hackbraten on Sunday December 01, 2019 @12:38PM (#21676649)
Did the original article not specifically make a point about Linux’s `rename` to be atomic only in the happy case and not when a crash happens during the rename?
Re:Files Are Hard (2015) by jng on Sunday December 01, 2019 @01:47PM (#21677080)
Rename is the most atomic you can get on Unix. Original article talks about partial file updates. New SQLITE file + rename should be as failproof as possible. And I’m any case nothing is 100% safe, so redundancy is a hard requirement for real safety.
Re:Files Are Hard (2015) by waterhouse on Sunday December 01, 2019 @02:41PM (#21677388)
I've encountered one gotcha there: SQLite operations may create "-journal" files, and you may have to be careful exactly what files you're copying or moving around.

More: https://sqlite.org/howtocorrupt.html , https://sqlite.org/tempfiles.html

Re:Files Are Hard (2015) by Hackbraten on Sunday December 01, 2019 @02:49PM (#21677422)
I was referring to the last section of the article, the part where it says rename isn’t atomic on crashes. Why not migrate traditionally via DDL in a transaction?
Re:Files Are Hard (2015) by mehrdadn on Sunday December 01, 2019 @04:11PM (#21677803)
It's not clear to me if they're correct about that. The way it's written, it seems to be saying "my read of the POSIX standard suggests rename may not be atomic on crashes", rather than "there are POSIX implementations in common use that have been observed to have non-atomic renames on crashes".
Re:Files Are Hard (2015) by nine_k on Sunday December 01, 2019 @02:35PM (#21677347)
On Windows you will at least have an old version and a new version lying around intact. Would take this over a single corrupted file any day.
Re:Files Are Hard (2015) by jasomill on Sunday December 01, 2019 @03:32PM (#21677626)
Unless you call FlushFileBuffers after all writes are finished, Windows makes no guarantee that the physical copy of a file on-disk will contain all previously written data, even if the file has been subsequently closed and renamed.

Typical (mis)behavior includes correctly-sized files that consist partially or entirely of zero bytes (because NTFS does guarantee newly-allocated space will be zero-filled).

Nevertheless, I agree with you, and, therefore, when dealing with application-managed files (as opposed to user-visible documents) on local NTFS filesystems, I typically

1. Write to a temporary file filename.temporary, say.

2. FlushFileBuffers

3. Rename the original file to e.g., filename.original.

4. Rename the temporary file to the original filename.

5. Delete filename.original.

Then, in some thread-safe manner at some point before accessing any such files — at application startup, say, when performance permits — I remove lingering ".temporary" files and restore any orphaned ".original" files to their original filenames.

This, of course, assumes FlushFileBuffers works properly, so, in particular, still fails in the presence of storage devices, drivers, and configurations (e.g., "Turn off Windows write-cache buffer flushing") that break buffer flushing, which are not nearly as rare as they should be.

Re:Files Are Hard (2015) by blattimwind on Sunday December 01, 2019 @08:20PM (#21679134)
> Unless you call FlushFileBuffers after all writes are finished, Windows makes no guarantee that the physical copy of a file on-disk will contain all previously written data, even if the file has been subsequently closed and renamed.

On *nix you need to sync before renaming as well.

Re:Files Are Hard (2015) by laurent123456 on Sunday December 01, 2019 @06:27PM (#21678525)
Do you have any example of what you're describing? I've used SQLite for years, written many migrations and never had such issues. A reasonably popular app I'm working on has so far 25 migration steps [0] and I've never encountered something like a new schema being different from an updated one. I'd expect whatever mistake was made to get to that point could be made in any other RDBMS.

0: https://github.com/laurent22/joplin/blob/805a5399b5288b4a282...

Re:Files Are Hard (2015) by cjfd on Monday December 02, 2019 @05:28AM (#21681289)
After writing the above comment, memory comes back a bit. The problem is that sqlite does not support renaming a column directly. One can find procedures for that on the internet. E.g., https://tableplus.com/blog/2018/04/sqlite-rename-a-column.ht.... However, we also wanted to be able to rename columns where there are foreign key references, for instance references to the table of which a column was renamend. A generic function, not in sql, but in the programming language of our project, in this case c++, was written for this purpose, something like rename_column(table_name, old_column_name, new_column_name). Needless to say the first version of this function was not quite sufficient leading to the problems that I mentioned. In the end I had to improve this function. I remember at the time I was quite pleased with myself figuring out how to do this without temporarily switching off foreign key checking which the first version of the script had.