News for hackers. Stuff that matters.
Re:SQLite as an Application File Format (2014) by ori_b on Saturday June 13, 2020 @12:23PM (#23510524)
I made a mistake -- it's an order of magnitude slower. That means a single transaction is within the threshold for human perception. If you block the UI thread on a transaction, you're now dropping 6 frames at 60fps. If you block other operations on the transaction completing, you end up with lag.

Reference: https://www.sqlite.org/faq.html, question 19. (I've seen similar when testing on SSDs locally).

Re:SQLite as an Application File Format (2014) by lostmyoldone on Saturday June 13, 2020 @12:52PM (#23510744)
If you have an FPS or interactivity target that needs to be guaranteed to be better than about 0.1-1 second - depending on platform - you can't have any disk IO at all on the UI thread, or any thread that's supposed to react at interactive rates at all times.

It has generally become much better in the last decade or two, but one should still expect most OS's to sometimes pause for excessive amounts of time on disk IO unless the API is specifically guaranteed to never pause. Even then one would be wise to measure/log deviations if it's critical for the application. OS guarantees might also be contingent on driver/subsystem guarantees, and bad drivers might sometimes affect what seems completely unrelated upstream systems.

Re:SQLite as an Application File Format (2014) by ori_b on Saturday June 13, 2020 @01:06PM (#23510865)
> If you have an FPS or interactivity target that needs to be guaranteed to be better than about 0.1-1 second - depending on platform - you can't have any disk IO at all on the UI thread, or any thread that's supposed to react at interactive rates at all times.

Yes, and other file formats encourage doing things in memory, so you don't have any disk i/o in the common path.

Using sqlite as a file format strongly discourages the simple, jank-free until you press save workflow of slurping your content, operating on it in memory, and then outputting it all in one operation as a response to an explicit user action. Instead, your whole application gets small but perceptible delays across all operations and interactions.

Re:SQLite as an Application File Format (2014) by lxdesk on Saturday June 13, 2020 @02:58PM (#23511746)
That conflates the use of the schema with the use of the format features. You can, in fact, use your custom in-memory structures and then "slurp it out" to BLOB when you're ready to save.
Re:SQLite as an Application File Format (2014) by ori_b on Saturday June 13, 2020 @03:02PM (#23511785)
Yes. But if you go that way, though, sqlite is just a slow and clunky format compared to, eg, json or protobuf. I doubt that any program out there uses sqlite this way, other than those which realize they painted themselves into a performance corner and hacked around it for compatibility.
Re:SQLite as an Application File Format (2014) by em500 on Saturday June 13, 2020 @01:57PM (#23511258)
Your concerns seem pretty theoretical. Several very widely used applications (Whatsapp, iMessage, Apple Notes/Calendar) use sqlite as a file format without much issue.
Re:SQLite as an Application File Format (2014) by ori_b on Saturday June 13, 2020 @08:42PM (#23514128)
It's also recommended against by Firefox because it's been causing performance issues: https://wiki.mozilla.org/Performance/Avoid_SQLite_In_Your_Ne...
Re:SQLite as an Application File Format (2014) by sdinsn on Saturday June 13, 2020 @09:56PM (#23514494)
It's funny you use that as an example, since Firefox itself uses SQLite as the application file format for history and bookmarks...
Re:SQLite as an Application File Format (2014) by ori_b on Sunday June 14, 2020 @01:04AM (#23515256)
Yep. if you read it, that page is basically their experience report with that decision.
Re:SQLite as an Application File Format (2014) by sdinsn on Sunday June 14, 2020 @09:21PM (#23522944)
> their experience report

It's not "their" experience report.

Re:SQLite as an Application File Format (2014) by ori_b on Monday June 15, 2020 @05:49PM (#23533420)
I see. Then how did it get on their site?
Re:SQLite as an Application File Format (2014) by lalaland1125 on Saturday June 13, 2020 @01:03PM (#23510836)
Why would you block the UI thread on a transaction?
Re:SQLite as an Application File Format (2014) by ori_b on Saturday June 13, 2020 @01:08PM (#23510880)
Because you've got another operation triggered by a UI action, and you need the previous one to finish, for example.
Re:SQLite as an Application File Format (2014) by yellowapple on Saturday June 13, 2020 @08:09PM (#23513977)
Sure, but you shouldn't need to block the thread for this. Most applications that have to deal with this sort of thing would present a loading indicator or something instead while a different thread is busy doing the actual transaction work. This is both possible and encouraged with (to my knowledge) literally every desktop or mobile UI framework written in the last 20 years.

And to be clear: this will still be user-visible with literally any other file format. The slow part ain't SQLite itself; it's the disk to which you're writing.

Re:SQLite as an Application File Format (2014) by sdinsn on Saturday June 13, 2020 @09:59PM (#23514509)
That doesn't answer the question. You never should have to block the UI thread. That's just poor application design, independent of the file format; you cannot blame that on SQLite.
Re:SQLite as an Application File Format (2014) by carlhjerpe on Wednesday June 17, 2020 @11:01AM (#23552225)
If the user interface depends on synchronous IO something is very wrong.