Things like cr-SQLite also have a lot of potential to make single SQLite per client a lot more viable. But I’m interested to see what you think the problems are? Have you found a solution or alternative?
Case in point of “engineers are not product people”
Back in the day, here is Sokovia, there was a local competetitor to Facebook. They had a great start and everything went perfect for them, but it quickly turned out that the technical side was really bad. Sluggish interface, constant outages, etc.
They tried to rewrite the app from scratch two times, and eventually failed.
So yes, making sure you're moving in the right direction at the beginning of your journey is pretty important. You don't have to overengineer and stay in your shed until you have a complete, feature complete product, but at least make sure, that you're building on the right foundation.
Just understand and accept that you are taking on heavy technical debt that will need to be repaid, and that it’s much more difficult to do once you’ve already vertically scaled several times along the way.
IME, many SaaS companies have eschewed the idea of having any DB experts, and this inevitably leads to pain down the road.
[0]: https://www.notion.so/blog/data-model-behind-notion
Unless they define, upfront, specific problems people really have, that their unified solution solves, then nobody is going to pay attention.
The second biggest problem is having an interface design team that makes all of those disparate apps consistent enough to be more usable than individual solutions. The fact that nearly no popular user-facing applications are developer-managed FOSS (as opposed to Firefox/blender/signal/et al which are managed by a company that hires professional designers) despite being free, tells you everything you need to know about dev-driven UI/UX. This is coming from someone that worked as a full time developer for years and contributed many thousands of hours of coding to FOSS projects before switching to design.
To me, rsync.net is peak design. It has just enough modernity to appeal to people who might expect that, but it quickly gets out of the way and tells you what it is, why it matters, and how much it costs.
At the other end of the spectrum, there’s tarsnap.com, which is probably a turnoff for anyone who doesn’t like text. I love it (as, apparently, do enough other people to keep its author comfortably employed), but I get that it’s an extremely narrow niche.
As a full-time developer for a decade, and in other technical roles for a decade before that, I had a few similar experiences with designers. One repeatedly insisted that Wordpress along with their ramshackle loopdy-looped spaghetti php plugin (still including comments from the tutorials they copied tidbits of code from) was robust enough to enough to replace our very tight Django-based code base that did a hell of a lot more than serve up our website… but they insisted it would take half as long to reimplement it all in php. There wasn’t even a good reason for it– they learned everything they knew about development by osmosis from working on web projects, and a mishmash of articles they read on the topic over the years, and after getting one piece of code to work in a low volume application, thought they were a dual-field specialist. That’s actually pretty rare among designers, but developers that feel that way about design are the norm. We all know what Larry Wall thought the three most important traits were for developers…
I wish we would develop software where the data model could easily change.
To do this every data dependency in the system needs to traceable. Nothing does this so far. And everyone just picks a database off the shelf but none are even remotely useful for this.
Unfortunately, at a very deep level, that’s simply not how RDBMS works. The tuples are a B+tree, and in some (MySQL [InnoDB], SQL Server) cases everything is clustered around the PK. If you don’t create a data model that’s easily exploitable for optimizations designed around that data structure, you’re gonna have a bad time. It’s no different than if you decided to use strings to store ints – you _can_, but it’s a bad idea for a variety of reasons.
What you can do is give yourself as much leeway as possible, by following some basic best practices. For example, it’s a hell of a lot easier to update a tiny reference table than to update billions of rows when you decide that column `region` should say `European Union` instead of `EU`.
My dream is to have a tool to model my logical data model and then it will organize my data into the best storage and caches.
I don't think any existing database today is useful.
Schemaless was one of the original drivers for NoSQL databases.
Now, when I need something schemaless, I start with a Postgres table with an ID and a jsonb or json field... which at least makes it easy to have a schema when the inevitable happens and schema-dependent code ends up getting added to the project.
> To do this every data dependency in the system needs to traceable.
This is a hard problem.
> Sync aPPS WItH OR WItHOUt tHE InTeRneT
Customers are going to store surprisingly large items in Docs, where you’d be tempted to inline them instead of offloading to S3 et al.
Chat practically needs to be its own DB. Discord runs on Scylla, Slack runs on Vitess over MySQL. The needs of chat access are wildly different from other types of storage.
If you’re doing any kind of active-active, have a plan for how to move off of that, because it does not scale (at least, not without breath-takingly expensive hardware).
Source: DBRE at one of your competitors.
EDIT: The fact that you’re doing offline saves (which is very cool!) makes me think that you may be using something like Ditto [0], which IIRC is MyRocksDB under the hood. I have no experience with either, but I do know some super sharp folks working at Ditto.
[0]: https://ditto.live