News for hackers. Stuff that matters.
Re:SQLite the only database you will ever need in most cases by olavgg on Thursday April 15, 2021 @03:42AM (#26817323)
I use Gitea(Github clone) locally with SQLite running on ZFS where I can take atomic snapshots of both the SQLite database and git repositories.

This would not be possible if I used a remote database, as there is no way to do a atomic snapshot of both the remote database on another filesystem(even if its ZFS) and the git repository.

Developers of multiple services, please support SQLite and don't lock yourself to a single database like PostgreSQL and MySQL. Even if the service is embedded in a Docker Container it is still painful to manage.

Re:SQLite the only database you will ever need in most cases by mulander on Thursday April 15, 2021 @03:52AM (#26817386)
> I use Gitea(Github clone) locally with SQLite running on ZFS where I can take atomic snapshots of both the SQLite database and git repositories.

ZFS snapshots may not result in a consistent databases backup of sqlite. You should use VACUUM INTO and then do a snapshot or use the sqlite Backup API.

See my other comment: https://www.sqlite.org/backup.html

Essentially, while you have a proper atomic snapshot of changes on disk the in flight transactions won't be there and you will be on the mercy of having a sucessful recovery from the journal/wal if you have that.

Re:SQLite the only database you will ever need in most cases by CodesInChaos on Thursday April 15, 2021 @04:30AM (#26817662)
When I back up a database, I expect to get a consistent snapshot of the database, which includes all committed transactions. Uncommitted transactions can't be included in this, since there is no guarantee they'd ever commit.

From your answer it sounds like I'd get such a snapshot (even if it relies on recovery from a journal internally). So I don't understand why you see a problem here. Is the recovery process unreliable?

Re:SQLite the only database you will ever need in most cases by OskarS on Thursday April 15, 2021 @06:26AM (#26818602)
Here's what SQLite has to say about the matter in their article "How To Corrupt An SQLite Database File" [0]:

> 1.2. Backup or restore while a transaction is active

> Systems that run automatic backups in the background might try to make a backup copy of an SQLite database file while it is in the middle of a transaction. The backup copy then might contain some old and some new content, and thus be corrupt.

> The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library. Failing that, it is safe to make a copy of an SQLite database file as long as there are no transactions in progress by any process. If the previous transaction failed, then it is important that any rollback journal (the -journal file) or write-ahead log (the -wal file) be copied together with the database file itself.

So no, it doesn't appear that it is safe in general to just copy the file whenever. If there are transactions in progress, things can go wrong. I don't quite understand why (isn't this the same as if a power failure happens, and SQLite is resistant to that?), but this is what the docs say.

It might be different with ZFS, since the backups are atomic (and this doc might assume that they are not), but I'm not 100% sure I would rely on it.

[0]: https://www.sqlite.org/howtocorrupt.html

Re:SQLite the only database you will ever need in most cases by CodesInChaos on Thursday April 15, 2021 @09:33AM (#26820248)
I think this warning only applies to non-atomic copies. Basic copy tools will copy a file piece-by-piece, where different pieces can have different age if the file is modified concurrently, which results in corrupt output.
Re:SQLite the only database you will ever need in most cases by mulander on Thursday April 15, 2021 @05:03AM (#26817957)
The only problem with atomic filesystem snapshots is the time required to perform recovery which can be avoided. The recovery process is of course reliable.
Re:SQLite the only database you will ever need in most cases by CodesInChaos on Thursday April 15, 2021 @05:26AM (#26818160)
Does recovery take a long time? Do you have any numbers?

I'd have expected recovery to take time proportional to the size of recent/uncommitted transactions, which should be quick, even for large databases.

Re:SQLite the only database you will ever need in most cases by mulander on Thursday April 15, 2021 @05:51AM (#26818339)
I don't have experience with large SQLite databases. I had PostgreSQL take over 40 minutes for recovery on a 2 TB database set to checkpoint every 30 minutes. How long it takes will depend on the WAL size, the frequency of checkpoints and the operations that were performed before a crash/snapshot.
Re:SQLite the only database you will ever need in most cases by Annatar on Thursday April 15, 2021 @05:37AM (#26818239)
That's not how database backup works: either you use a tool like RMAN, which knows about internal Oracle database structures, or you must shut a database down in order to do a cold backup. As there is no such thing as database shutdown in SQLite's case, if you want a clean backup before doing a ZFS snapshot (and shipping it off, else it's not a backup), you must shut down the application which is writing to the SQLite database.
Re:SQLite the only database you will ever need in most cases by CodesInChaos on Thursday April 15, 2021 @09:40AM (#26820348)
Since power-loss is (hopefully) equivalent to an atomic storage snapshot, a database which doesn't produce a consistent backup via snapshots can't be safe against power loss and thus lacks the durability property of ACID.

What's unsafe is using a naive file copy tool (e.g. `cp`), which non atomically copies a running database.

Re:SQLite the only database you will ever need in most cases by Annatar on Thursday April 22, 2021 @05:24AM (#26900598)
Without taking the aforementioned steps I specified, power loss is not an atomic event, because most databases nowadays rely on the fsync(2) system call to tell them when the I/O has completed. If the fsync(2) call is unreliable because it is lying about completing the requested I/O, the RDBMS stands no chance of guaranteeing atomicity. I am sorry.
Re:SQLite the only database you will ever need in most cases by jeltz on Thursday April 15, 2021 @04:44AM (#26817791)
Are you sure? Since for PostgreSQL an atomic snapshot is enough. It is only when you do not have access to atomic snapshots that you strictly need to use the online backup api. And, yes, it will have to do a crash recovery when you first use the backup but if you cannot trust crash recovery you can't really trust your database at all.
Re:SQLite the only database you will ever need in most cases by mulander on Thursday April 15, 2021 @05:00AM (#26817928)
You are correct and what I had in mind was the required recovery. I do trust recovery but performing it can be very time consuming.
Re:SQLite the only database you will ever need in most cases by 7steps2much on Thursday April 15, 2021 @04:07AM (#26817488)
It is important to mention that a "Remote" database could run on the same host ...

I run PostgreSQL on my VPS alongside gitea and some other services, and I can take snapshots of it just fine (assuming I don't take a snapshot as it is writing to disk, but that problem exists with SQLite as well).

Re:SQLite the only database you will ever need in most cases by CodesInChaos on Thursday April 15, 2021 @04:25AM (#26817619)
1. You can run Postgres locally using the same ZFS for storage you'd use for the sqlite database. This should come with the same backup properties as sqlite.

2. I doubt that the git part of your setup can be snapshotted correctly while an operation is in progress, since git doesn't use transactions.

Re:SQLite the only database you will ever need in most cases by theamk on Friday April 16, 2021 @12:02AM (#26829312)
Git does use transactions, in a pretty clever way: the main storage is immutable -- so when you do a new commit, it creates all new files, and as a last step, updates HEAD/branch, using "rename" call which is atomic on all filesystems.

So you can do pretty bad things to git and it would still work. Atomic snapshots are absolutely no problem. Copying the directory while in progress is fine.

If you try very hard, you can still make a bad backup -- say you were copying files from remote host, and backup got interrupted after HEAD but before the new data. In this case, there is reflog -- append-only file which keeps HEAD history. Using this file, you would be able to manually recover into previous state.

Re:SQLite the only database you will ever need in most cases by croon on Thursday April 15, 2021 @03:48AM (#26817362)
I've deployed Bitbucket and Postgres as a docker stack under the same subvolume and done snapshots on that. As long as you have WAL enabled in Postgres there shouldn't be an issue, and I've restored multiple times without problems.