S3QL: an S3 FS with encryption, de-dup, immutable trees and snapshotting

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by nbevans on Tuesday September 01, 2015 @05:00AM (#10151370)

SQLite supports multiple readers when Write Ahead Logging (WAL) is enabled. This mode is so good and transforms the whole operation of SQLite that it is shame that 1) it isn't the default, 2) more people don't know about it.

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by edwintorok on Tuesday September 01, 2015 @12:05PM (#10153361)

There is also `PRAGMA synchronous = NORMAL` that helps a lot. Your database will still be consistent and cannot become corrupted, although you do have to give up transaction durabibility after a power loss:

https://www.sqlite.org/pragma.html#pragma_synchronous

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by aidos on Tuesday September 01, 2015 @05:31AM (#10151458)

The way s3ql works is that it copies the db from s3 to your local machine. You make changes and then write the new version back to s3 later with the data. So it won't help in this case.

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by nbevans on Tuesday September 01, 2015 @06:03AM (#10151517)

I see. Out of interest how does it write back the new version of the SQLite database file with only the differences? Does S3 support binary patching?

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by aidos on Tuesday September 01, 2015 @06:19AM (#10151557)

I believe it uploads the whole thing each time. I think it may even upload it with a new index counter in the name to version it (but I can't find that in the docs now).

http://www.rath.org/s3ql-docs/impl_details.html#metadata-sto...

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by tobias3 on Tuesday September 01, 2015 @07:19AM (#10151687)

I think you found the Achilles heel. This won't scale, so it is only suitable for small file systems.

[ Parent | on HN ]

Re:S3QL: an S3 FS with encryption, de-dup, immutable trees a... by nbevans on Tuesday September 01, 2015 @07:35AM (#10151718)

Indeed, I'm good at that ;)

It could probably be improved (subject to the nuances of S3 which I'm not fully familiar with). One way to fix it would be to copy the concept of SQLite's WAL mode. Use an appended write operation on S3 (if it supports it) to append to an existing file that contains the transaction log. Then at certain intervals (say every few thousand transactions) one can finally flush that log to be stored in the main database file.

This would substantially reduce the number of times the database would need to be re-uploaded in full.

[ Parent | on HN ]