News for hackers. Stuff that matters.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by belter on Wednesday June 12, 2024 @06:27AM (#40656495)
In 2024, not using SQLite as your application file format, should really force an engineering team to have a damn good explanation....

https://www.sqlite.org/famous.html

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by fauigerzigerk on Wednesday June 12, 2024 @08:00AM (#40657079)
Those journal/wal files create a number of potential issues for user visible application files. E.g. a user could copy a database file into a directory that already has an old journal file, corrupting the database [1].

Or users could simply get confused about what to do with those files and contact support a million times just to ask why these files exist, whether they need to be copied as well, why they are getting large, etc.

These issues can be worked around, but I wonder how many apps that use SQLite actually bother doing that. Clearly not all of them [2].

Avoiding file corruption is non-trivial regardless of file format, but giving users additional ways to corrupt their data is never a good thing.

So I think for small, user visible application files that don't need any database functionality, the onus is still on engineers to justify their choice of SQLite.

[1] https://www.sqlite.org/howtocorrupt.html

[2] https://forum.audacityteam.org/t/aup3-wal-file-remains-and-l...

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by wil421 on Wednesday June 12, 2024 @08:20AM (#40657249)
>Or users could simply get confused about what to do with those files and contact support a million times just to ask why these files exist, whether they need to be copied as well, why they are getting large, etc.

This reminds me of a college roommate who was dissatisfied with his computer, likely due to malware from sketchy sites. He decided to delete large or suspicious files from window’s system32 folder. By suspicious I mean he didn’t like the file name.

He had to get a new computer later in the semester.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by macintux on Wednesday June 12, 2024 @08:23AM (#40657270)
Decades ago I troubleshot a Solaris server that wouldn’t boot, and discovered the kernel file was missing. I suspect someone had tried to deal with a too-small root partition by removing large files.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by eli on Wednesday June 12, 2024 @12:47PM (#40660188)
You could simply turn off WAL for a simple application database, if you were really worried. You lose some performance and scale, but it should work fine.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by fauigerzigerk on Wednesday June 12, 2024 @03:00PM (#40661676)
Setting the journal_mode to OFF or MEMORY is a sure-fire way of corrupting the database. It's not fine.

https://sqlite.org/pragma.html#pragma_journal_mode

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by eli on Wednesday June 12, 2024 @03:42PM (#40662160)
If the application crashes in the middle of a writing a transaction, yes. You can work around this if you care to, and I don't think it's anything specific to SQLite. What happens if your application crashes in the middle writing out a JSON file?

I had actually meant JOURNAL_MODE=DELETE, though. This would not technically use a WAL file, though there would be a rollback journal file that exists only for the duration of the transaction.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by fauigerzigerk on Wednesday June 12, 2024 @04:28PM (#40662721)
>What happens if your application crashes in the middle writing out a JSON file?

The usual way to deal with this is to write to a temporary file and then atomically replace the original file with that temporary file. You could do the same thing with an SQLite database. This is the workaround I was referring to earlier.

journal_mode=delete may not be the worst compromise, but I hate the idea of exposing users to potential database corruption, even in a relatively unlikely event.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by renonce on Wednesday June 12, 2024 @02:42PM (#40661436)
> 1.4. Mispairing database files and hot journals

I’m surprised it would corrupt the database. The database could have included a unique identifier as well as in the journal and refuse opening the database if the identifiers don’t match.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by fauigerzigerk on Wednesday June 12, 2024 @03:14PM (#40661839)
Uniquely identifying a database and its journal/wal files would prevent just one of the five mispairing cases they mention. And it's the least likely case.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by throwup238 on Wednesday June 12, 2024 @06:59AM (#40656658)
> The United States Library of Congress recognizes SQLite as a recommended storage format for preservation of digital content.

That’s one hell of an endorsement.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by ecuaflo on Wednesday June 12, 2024 @07:51AM (#40657003)
unfortunately in the browser it needs to be downloaded so you’re stuck with indexeddb
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by sgbeal on Wednesday June 12, 2024 @11:11AM (#40659004)
> unfortunately in the browser it needs to be downloaded so you’re stuck with indexeddb

That argument implies that we're stuck with the browser's single built-in home page because every other web page has to be downloaded.

How is having to download sqlite3.wasm any different from having to download HTML, CSS, JS, images, etc.?

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by w0m on Wednesday June 12, 2024 @11:23AM (#40659141)
> How is having to download sqlite3.wasm any different from

'unnecessary' file size. K matter on the web still.

I'm not normally a web guy, but from https://sqlite.org/download.html it looks like it's ~800k extra added to initial page load? Based on average mobile speed in US of 97.09Mbps; that's an extra half second added to initial page load. That's not trivial; maybe an extra 15% bounce rate on initial hits.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by sgbeal on Wednesday June 12, 2024 @11:37AM (#40659300)
> ~800k extra added to initial page load?

If you're serving it uncompressed, which no production-grade site will (for a given definition of "no"/"none").

> K matter on the web still.

Not, i opine, for the types of apps which want to host client-side databases. These are client-side applications, not "web pages."

Even a bare-bones, database-less google.com is now 3.14mb uncompressed (1.32 compressed), and that's not counting the pieces which uBlock Origin keep from loading. It loads somewhere around 2MB (uncompressed) of JS.

Last i checked, gdrive downloaded some 14mb to get up and running.

> that's an extra half second added to initial page load. That's not trivial;

We'll have to agree to disagree on whether half a second extra initial-hit-only load time is trivial.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by fwip on Wednesday June 12, 2024 @01:02PM (#40660371)
> If you're serving it uncompressed

That 847 KB is actually the size of the (compressed) zip file, but it does contain other files. The compressed WASM + the JS to load it is probably about 500K, depending on the specifics of compression & minification.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by Digit-Al on Wednesday June 12, 2024 @02:00PM (#40661016)
Makes me laugh. When I got my first computer in 1987 it took so long to load a small game from cassette that I could go downstairs, boil a kettle, make a cuppa, and bring it back to my room, and the game would be just about finished loading. Now people are moaning about an extra half a second lol

How things change.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by paulddraper on Wednesday June 12, 2024 @02:32PM (#40661317)
No, the narrative here is that things used to be fast and idiot JS devs have made everything bloated and slow.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by andrepd on Wednesday June 12, 2024 @07:35PM (#40664344)
Computers are thousands to millions of times faster. So yes, that's a true assessment.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by paulddraper on Wednesday June 12, 2024 @08:37PM (#40664727)
> millions of times faster

Unless you had a 3KHz processor (and no processor ever had that), that statement is just not true.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by andrepd on Friday June 14, 2024 @03:04AM (#40678277)
It's not about raw clock speed, or a Zen 4 would be as fast as a Pentium 4 :)
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by jacobprall on Wednesday June 12, 2024 @05:07PM (#40663143)
It's a matter of use case - for simple web page or even a lightweight client-side app, maybe too big. But for tool with heavy data processing, I could see it being quite useful.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by iansinnott on Wednesday June 12, 2024 @11:17AM (#40659086)
It's too bad WebSQL never took off, as that was sqlite built in to the browser.

From memory the wasm binary of wa-sqlite is ~1mb, which is certainly not nothing but is an acceptable one-time download size for many web apps. I've seen websites with single image files larger than that. Not an advocating for more bloated web apps, but a sqlite download might not be a deal breaker.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by threatofrain on Wednesday June 12, 2024 @07:13AM (#40656746)
Is SQLite the best solution for on-device logs?
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by eternityforest on Thursday June 13, 2024 @01:35AM (#40666282)
There's one big obvious one. Git and SyncThing.

If the user could benefit from either of those, atomically updated plain files are great. Otherwise, SQLite is perfect.

Of course there's always workarounds to use SQLite in memory and dump to SQL code and such.