News for hackers. Stuff that matters.
Re:SQLite is all you need for durable workflows by petre on Saturday May 30, 2026 @12:40AM (#48332649)
We recently just partitioned the data into many SQLite databases and got away with it. It's telemetry data from IoT devices: one device, one database. Backups are an easy rsync job now instead of streaming a multi gigabyte database with compression that take hours. Reporting will just open each database and aggregate multi device data into another database (Duckdb, SQLite or something else, we'll see). Duckdb is not readable when locked so it's probably also going to be SQLite. Even it it's going to spit out JSON it will go into SQLite rows instead of many files.
Re:SQLite is all you need for durable workflows by datadrivenangel on Saturday May 30, 2026 @01:14AM (#48332827)
Check out Quack for DuckDB.
Re:SQLite is all you need for durable workflows by petre on Monday June 01, 2026 @06:31AM (#48354951)
I saw that but it's a bit too much friction. We'd rather attach a SQLite database and copy the results into that. Also no worries about version changes and compatibility, the types are not too restrictive. But we'll see.