Bonus points for some open standard, but some kind of understandable format even if proprietary is miles ahead of proprietary and cryptic.
I wish Fusion360 for example had a sanely git-able on-disk format like:
line 0xdeadbeef (0,0) - (123,456)
line 0x01234567
constraint parallel 0xdeadbeef 0x01234567
Etc., Or something.(Actually I suppose git - and incremental backups - is the main reason I say I'd still prefer plaintext to db.)
Everything today uses and understands ASCII/UTF-8 and we've used that, plus some handy human conventions like limited character count between newline characters, to build up tools like git to help manage the changes in smaller chunks.
There's nothing preventing us from doing the same for other formats if they become commonly supported or expected. Much like SQLite and some of the tools cropping up.
But ubiquity is extremely special and useful and essential if you want to maintain ownership of your data.
True. I mean if it would be common to put source code and text into sqlite files, then most tools would probably also deal with it. However there is a case about simplicity. For instance writing a diff algorithm on plain text files is much simpler and easy to understand, than writing diff algorithms on text or binary structures.
I'd argue the opposite. Coming up with a good, human readable, diff on unstructured text within acceptable runtime is not that easy. LCS is NP-hard and the naive algorithm has unacceptable performance and memory footprint.
However diffing something structured, especially when it has unique keys like in most databases, is really easy. Structure gives you an easy way to compare data systematically and is handy to express differences.
An Sqlite database may be a binary blob, but thanks to the Sqlite software being open source and ubiquitous, it's pretty easily comprehensible. Sure, not quite as accessible as plain text[1], but not far from it.
So when you need something just a little bit more advanced than plain text (i.e. SQL functionality) it seems like a good compromise that doesn't -- pun intended -- too much compromise openness and accessibility.
___
[1]: But, hey, does "plain" text include UTF-8 nowadays or does it still mean ASCII-only?
There are two standards for this common to CAD, called IGES and STEP. They are both plaintext, though their gittability is questionable.
Is there a non-proprietary somewhat sane way to do version control for spreadsheets (Excel and LibreOffice)?
That is not exactly what proprietary means. Though I feel the same - plain text can be versioned and managed by various tools and that is great!
Plaintext could be in a proprietary format but that's far better than proprietary compressed database that's a job to work out what even made it (if that wasn't proprietary itself!). Proprietary plaintext might even be better than open standard blobby format, depending on what you want to do with it. (Better for git and backup, worse for opening in competitor tools.)
If your code is Free and Open Source, then its file-format is presumably not proprietary, whether or not it makes use of an existing standard format (like JSON) or solution (like SQLite).
Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead.
Do you not need reliability, integrity, security, performance? Go ahead.
So use a file to store data instead of database but know the trade-offs.
As soon as you start processing forms and doing anything transactional based on that, you'll be quickly reinventing a lot of wheels if you aren't using a DBMS of some sort.
Doesn't Hacker News run on flat files?
And if you e.g. have multiple dataflow paths with different priorities then you also probably shouldn't be using a DBMS, because they're too much of a closed world. Using database technology as a library would be a better approach; otherwise sooner or later you'll find you need to customize the internals of your DBMS, and with most extant products that's somewhere between hard and impossible.
Could you expand? Does this mean you wouldn't use an ACID database for CRUD?
99% of the time all you want to do is a) commit input events to a durable store as quickly as possible b) read a state-of-the-world that's consistent in the sense that it represents all the consequences of all the inputs up until some (logical) time t, and IME an event sourcing model is if anything better at doing that than an RDBMS is.
A database has a whole engine, which I'd argue is a pretty fundamental difference.
I think I get what you were trying to say, though: there's no fundamental difference between database tables and flat files, and there's no fundamental difference between a database and a filesystem.
But "flat" essentially means "one table that doesn't reference (and isn't referenced by) other tables in a reliable way." It's a way of saying 2 dimensions (flat) versus 3+ dimensions (relational). The moment you start having multiple plain text files where columns act as primary and foreign keys, the system is no longer flat, despite using plain text files.
It seems to me that TFA and most comments here are really just talking about using a DBMS vs using plain text files, not using flat vs using relations.
You can tar up your flat files as well, but that is a separate process and can be time consuming for larger directories.
Perhaps there is a use case for a file system backed by portable database like sqlite, such that you can just mount the volume and send it around without pain, but can prefer to use normal command line utils (ls, rm, grep, etc) instead of interacting via SQL.
(This also works without Docker of course, but Docker makes me feel safer when I do stuff like this)
Since we're talking about copying a folder, I'm not sure why you'd feel safer doing it via an additional layer of abstraction.
I know that PG data is usually in /var/lib/postgresql/data, but I like Docker because if I ever need a 2nd or 3rd database on one system, I can give them more meaningful names and not feel like I'm touching Important System-Wide Stuff.
My intuition is that you're expected to do some kind of migration or export/import, because the on-disk format probably changes between major versions.
We have file formats like ODF which are just zip packages with regular files inside
DBMS (and in particular) RDBMS is not a merely persistence medium for an application - DBMS main role is to _share_ data between applications.
Curious as to what is not just "all bytes on a disk/memory in the end"? Also can we please stop calling datastores databases?
There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software.
For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be necessary with fast SSDs. If your reason for splitting into small files is to save RAM, it may not be necessary if you have more RAM. If you want to do to a distributed architecture, maybe your filesystem is not up to the task and you may want a database server.