So the assumption that "if anybody changes anything everybody needs to know it" is close to true. In that scenario, putting a bunch of APIs in the way just makes the data less coherent rather than more. In most other scenarios, yeah, it's hard to see that it really makes sense.
In what use case do you run into that?
I'm lead and an architect on an enterprise application at the moment that drives the whole company. It's your standard configuration, front-end, APIs, SQL. The system requests what it needs to fulfill only what functionality the user is dealing with.
Earlier in my career I was dealing with large enterprise desktop applications that talked directly to the database, with no centralized API. Some of them had thousands of individual desktop clients hitting a single multi-tenant SQL server. No problem, SQL Server would handle it without breaking a sweat. The bandwidth to an indidual client was fine. It was fast. And that was 20 years ago.
- try to reduce the amount of paper based catalog we were sending out to customers, those were a couple thousands of pages and not cheap to produce, not cheap to send and would get deprecated very quickly. The web app would be pulling the entire catalog at first load so customers could go in remote location and still be able to use the catalog
- a web app for sales that was intended to be use on customer site containing all the marketing materials and much more during presentations on site without ever having to connect anywhere
Single-tenant simple sites without permission checks with limited content volume.
Of course you can also do the same in a multi-tenant environment but naturally you wouldn’t be returning all rows in the database.
User can open the application, get the database, the frontend does all the offline updates the user want to perform. The user can update their website, add bookings they received on the phone, check what prices they set. This is all blazingly fast with no loading time, no matter your internet connection, because no further communication with the server is needed.
At some point they press a magic Publish button and the database gets synced with upstream and uploaded. The upstream service can take the data and publish a website for its users, update availabilities, update prices, etc.
It would be a better user experience than 99% of the SaaS out there.
You might remember horror stories from the late 1900s when people would lose critical data and documents because they "forgot to hit Save and the computer crashed." Or, maybe you've never experienced this — because a couple decades of distributed-systems engineers and UX researchers made that problem obsolete.
So now we've... reinvented the Save button, but somehow needed a Paxos implementation to do it?
Everything old is new again, I guess.
I want to decide when to save, and how, not the application. I am not a product, I am a user!!!
If I open a word document, redact a few bits, save to PDF and close it - I don't want my changes saved but they are (real scenario from last week. I'd only just moved the file into Onedrive so autosave had turned itself on)
Ditto sorting and filtering spreadsheets.
Software engineers: By all means save in the background so I never lose anything, but don't assume I want the latest changes.
This is not trivial to implement, though. Complexity will depend a lot of the application and the data. I'd say it'll only make sense for mature apps. A startup will most likely don't consider this in its roadmap.
The thing that I'm unclear on is how do I figure out what data to ship to the user? Like if I don't already have a database-per-user then I have to extract only the data the user has permission to see, and ship that as a database?
That would be the case even if I had database-per-customer - not every user is necessarily able to see all the data owned by the organization they're a part of.
It seems like a lot of extra work, and error-prone, too (what could be worse than accidentally shipping the wrong data _as an operational database_ to a client?)
Edit: the article covers this at the bottom, but IMO it's a show-stopper. What application of any real complexity can actually implement database-per-user (database-per-tenant is probably not enough, as I mentioned above).
Now, you'll introduce a huge amount of user frustration around why their changes never made it to the central database. If you have users closing a form, especially if it's on a webpage, they are going to expect the write to happen at the same time the form closes. Making a bunch of changes and batching them in a single sync is going to be confusing to a ton of users.
I just checked a heavy user for our enterprise SPA. localStorage is using ~12 KB.
I don't know what people are just throwing in localStorage but they certainly aren't pulling down much of the database (which still would have to be checked to ensure cached data in there isn't stale).
"get the database"
How small do you think these databases are?!
You're going to download the entire hotel booking platform's database?
For how many hotels? One at a time, and then get another databse? Or are you getting a Sqlite booking database for every hotel in the world? And you're going to send them to each user? For what date range?
And even if that were possible, you then have to commit your offline updates. What if someone else booked the date range prior to you? Now your Sqlite copy is stale. Download the entire thing again? There could have been countless changes from other users in the time since you last got your copy.
This explanation just leaves me even more confused. It's illogical.
Jokes aside, this extreme optimization for development does have impacts on user experience. The amount of bandwidth/storage etc used by a "just ship the whole db" type applications would surely suck for most people outside of the 'I measure my bandwidth in Gbps' crowd?
For this project, I'd do:
* A database for each user (this is natural on CouchDB, a setting one can enable to create the DB automatically. The platform won't choke on many databases) - for some per-user data, like comments, if the user has already reserved a booking we can mirror booking data there + some info regarding the hotel.
* Common synced databases - some general info regarding the platform and hotels. Preferably not too large (IIRC there's no limit to attachments with 3.x, but it sounds risky). Anything too large we'd do online or use data provided with the application.
* A large booking database which isn't synced and must be modified online with a REST call - we don't have to allow every action offline. This obviously needs the online component for making sure bookings don't conflict. Could even be a regular relational database. (Something like allowing a provisional offline reservation to the user database followed by a possible rejection on the server when syncing is doable, but I'm don't think there's much value to the user here).
It sounds very doable, rather similar to a project I was involved with, but I miss SQL a lot with that platform. A sqlite-based approach is an interesting alternative.
It doesn't sound that unreasonable.
Even if you have 3-4 hotels your data won't be significant.
By the time your dude downloads the database half the hotels on your imaginary website have changed status.
1. Internal tooling: you're able to manage the contract better since you have control of the source & destination applications.
2. Reporting & analytics: these tend to need a lot of query flexibility & they tend to use a lot of resources. Offloading the query computation to the client makes it easier on the source application.
As for database size, the Corrosion program mentioned in the post is about 8GB and has a continuous write load so this doesn't have to just be for tiny databases.
Basically, the solution they have is:
1. There is a single writer. There's optional best-effort leader election for the writer.
2. If there's a network partition, split-brain, etc, availability is chosen over consistency.
Jepson's testing is focused on databases that pick "consistency". Since LiteFS didn't pick consistency, there's really not any point in running Jepson against it. Like, jepson would immediately find "you can lose acknowledged writes", and LiteFS would say "Yes! That's working exactly as intended!"
However, another way of running LiteFS is with only a single writer ever (as in one app, one server, one sqlite database only), and all clients as read-only replicas that are not eligible for taking writes ever. In that case, you also don't have a proper distributed system, just read only replicas, which is quite easy to get right, and mostly what this post is talking about.
Fly is not a database company.
The fact that a company like Fly routinely shows up on HN for their services being down boggles the mind that anyone would even think about using their "distributed database" bullshit.
Quite insane times the 2023s are.
On the LiteFS Cloud side, it currently does streaming backups so it has similar guarantees but we are expanding its feature set and I could see running Jepsen testing on that in the future. We worked with Kyle Kingsbury in the past on some distributed systems challenges[1] and he was awesome to work with. Would definitely love to engage with him again.
I don’t have a horse in this race the reply isn’t very well I don’t know what to make of that
Some might be using Postgres or whatever the Engineering team provided them with, but I don't think I have really heard of a Data person preferring to use SQL Lite.
Might still be a great fit, but as the other comment pointed out, it might not be a good fit for the target audience.
Are there any plans for Corrosion to be published as OSS?
> 1. Internal tooling: you're able to manage the contract better since you have control of the source & destination applications.
This has not been my experience in anything but tiny companies or companies with very strict mono-repo processes. One big point of separate teams is to minimize the communication overhead as your organization grows (otherwise it scales as N factorial). That means you do not want a lots of inter-department dependencies due to the internal tooling and APIs they leverage.
> 2. Reporting & analytics: these tend to need a lot of query flexibility & they tend to use a lot of resources. Offloading the query computation to the client makes it easier on the source application.
Depends on the resources the client has versus the server to devote to a single query. The resources are also high because of how much data is analyzed and 8gb seems tiny to me (ie: the whole thing can be kept in some DBs memory).
They are adding a sync-mechanism for Sqlite so local writes can be synced to a remote location and on into Postgres and so on. CRDT-based eventual consistency.
So local write latency, eventually consistent sync and the tools for partial sync and user access are the primary areas of development.
Early days but an interesting approach to shipping the db.
I replicate the clients Sqlite DB to a central server (over a REST-API) where it is synced with existing data. I use an CRDT, so changes don't get lost and as long as the clocks of all the users devices are accurate, the merges are in the correct order.
This enables offline access, but also the use of my app without an account. You can always merge the data later. Multi-device merge is also possible, if you create an account. Especially the multi-device merge was a big headache until I settled for this approach.
Since it is a home-grown solution I still have to do some manual stuff that could be abstracted away, like converting datatypes to and from JSON, Kotlin, PHP, MySQL. There's not always an equivalent datatype in those technologies.
This approach probably won't work well for big databases that share data between multiple users, though.
It was not uncommon in the early 2000's for applications targeting mobile professionals and sales types to work this way, except that instead of SQLite they used things like Notes¹, Jet² and MSDE³. By modern standards these would be considered "tiny mostly static databases" often not exceeding a gigabyte. Instead of connecting over the internet they used serial dial-up file transfer protocols to synchronize with a central database or a mainframe. People would typically download information in the morning, make updates offline and synchronize at the end of the day. A slow trickle of periodic bidirectional updates insured everyone had a reasonably fresh copy of the information they needed.
1- https://en.wikipedia.org/wiki/HCL_Domino
I mean, I get it, from a technical standpoint. Ok, so you're going to send read-only Sqlite databases to everybody.
Is it missing what the API (that you still need) is updating when you insert or update something and all client DBs are now stale? Is there a central database? How often are you pushing out read-only database replicas across the wire to all clients? Is that really less "chatty"? If so, how much bandwidth is that saving to push an entire database multiplied by the number of clients?
None of this seems logical. Maybe I'm missing the real-world use-case. Are we discussing tiny Sqlite database that are essentially static? Because in the last 30 years I've not run into a situation where I needed to have clients execute SQL queries on tiny, static databases let alone still need to potentially update them also.