LiteFS Cloud: Distributed SQLite with Managed Backups

Re:LiteFS Cloud: Distributed SQLite with Managed Backups by Spunkie on Wednesday July 05, 2023 @01:58PM (#36604335)

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream and scripts for too long.

I've been bugging cloudflare to do this with D1+tunnels since D1 was announced and they constantly seemed confused what I was even talking about.

[ Parent | on HN ]

Re:LiteFS Cloud: Distributed SQLite with Managed Backups by benbjohnson on Wednesday July 05, 2023 @02:09PM (#36604556)

Let me know how it works for you if you get a chance to try it out. We learned a lot since making Litestream so hopefully it's a big step up! The underlying format of how we're storing snapshots and incremental backups should make restores A LOT faster. Litestream has to replay all WAL writes since the last snapshot whereas LiteFS is able to perform a lot of deduplication and use multi-level compaction to create a minimal set of restore data.

[ Parent | on HN ]

Re:LiteFS Cloud: Distributed SQLite with Managed Backups by jgrahamc on Wednesday July 05, 2023 @02:01PM (#36604377)

Email me and tell me [email protected]

[ Parent | on HN ]

Re:LiteFS Cloud: Distributed SQLite with Managed Backups by bjyule on Wednesday July 05, 2023 @07:11PM (#36608525)

What do you have in mind with respect to D1+Tunnels?

We're launching PITR support for the new experimental backend within the next 1-2 weeks. Stay tuned.

(I am the eng director for Cloudflare Workers)

[ Parent | on HN ]

Re:LiteFS Cloud: Distributed SQLite with Managed Backups by Spunkie on Saturday July 15, 2023 @01:39PM (#36739236)

I wanted this when D1 was first teased, and before I figured out that D1 is a database product that happens to be built on SQLite rather than SQLite being the star of the D1 show. Nonetheless, what I want is to be able to point a CF tunnel at a local SQLite file and instantly be able to leverage all the benefits of D1.

I should be able to...

    1. Configure a CF tunnel through the config file or the dash, pointing it to a local path of an existing SQLite file.

        1a. On startup, the SQLite file should create and link a new D1 db.

    2. Configure a CF tunnel through the config file or the dash, pointing it to an existing D1 db.

        2a. On startup, the local sqlite db should be either overwritten or created.

    3. Create read replicates through CF tunnels on another server or the local computer/wrangler for development.

    4. Have automatic global read replicas on the edge, which is inherent to D1 no?

    5. Have automatic/rolling backups and export and import those backups automatically with S3/R2.

    6. Do a PITR through the dash or CF tunnel CLI.

        6a. PITR should be possible to either the existing live DB or restoring it into a separate new D1 copy.

There are likely plenty of more cool things that could be done if D1 could be exposed as a normal SQLite file. Ultimately, CF tunnels are just in the mix because it seems like the obvious choice for plumbing SQLite files into and out of the CF network/edge.

[ Parent | on HN ]