> In 2022, we shouldn't be reserving any area of a file as 'spare space' like this.

In 2022, we should have file formats that are not naive of the underlying filesystem. I store my text notes in Git, but I also have two LibreOffice documents in there because org-mode doesn't handle their contents well. It would be nice of only 5% of the file had to be rewritten when I change a single word, rather than the current 91% of the file. A 5% hit to storage for a 91% reduction each time I update it would be a great trade off.

Under the hood, odt and docx files are zip files.[0] The “91%” change has nothing to do with rewriting the file; it mostly has to do with zip files and how poorly git handles binary diffing. Plaintext files are rewritten on disk every time you save them in virtually every editor and that doesn’t cause problems for git.

I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again.

Alternatively, you could just switch to a text-based format like rtf, as long as you don’t need any specific features from odt or docx.

(EDIT: someone else mentions a promising sounding “flat xml” format, which I’ve never encountered before.)

[0]: https://superuser.com/a/1356829

>I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again.

I have a current project where one of the other developers has tried this. It might work for a solo project, but with multiple developers on Windows (Excel) and Linux (Open Office), I find that Open Office feels a need to continually fuck with unrelated fields. Editing a single field in an xslx file results in changes in a dozen files in the unpacked version. Making any more complicated changes results in a diff that's impossible to manage.

Do you actually mean OpenOffice, or do you mean LibreOffice? I would hope everyone is using the much more improved LibreOffice by now instead of OpenOffice.

Still, interesting to hear some feedback on how well (or not) that idea has worked for some.

The Flat XML option seems to be the best approach if you need to edit documents more complex than what can be represented in either markdown or rtf (rich text format).

In your case, LibreOffice allows you to save the file as Flat XML (e.g., .fodt), and the document will be written as a single uncompressed file. Unfortunately, few apps have a feature like this.

This is perfect for my use case, thank you! The .fodt file is 150K, vs 117K for the .odt file. But after changing a word less than 20 of the 1983 lines in the file have changed.

  $ ls -hl todo.fodt
  -rw-rw-r-- 1 dotancohen dotancohen 150K Apr  7 16:41 todo.fodt
  $ ls -hl todo.odt
  -rwxrwxr-x 1 dotancohen dotancohen 117K Apr  6 12:01 todo.odt
  $ wc -l todo.fodt
  1983 todo.fodt
  $ cp todo.fodt todo-edit-one-line.fodt
  $ libreoffice todo-edit-one-line.fodt
  $ diff todo.fodt todo-2.fodt | grep "+ " | wc -l
  19

Wow, thanks for the Flat OpenDocument file type! Such kind of files are much better to treat with git (i.e. human readable diffs, for instance). However, unfortunately, these files tend to get very large (200kB ODS vs. 7MB FODS). What a pity the XML itself is not less verbose.

I dunno if that will help. I know Dia lets you store the uncompressed XML, but I have seen it sometimes have a 100% diff due to the serialiser moving *everything* around when all that happened was that I rearranged half the blocks on the diagram.

One option is to store files as a database like SQLite, which I assume is designed around not rewriting the whole file on every change. For example Audacity 3.x does this.

However SQLite stores multiple files on disk for crash persistence, complicating matters. But then again so does Audacity 2, and Microsoft Office to an extent (file locking rather than data integrity).

Microsoft Word used to only append changes to the document on save rather than rewriting the whole file, but this was disabled since if you deleted text and saved the file, it remained in the document. https://www.cnet.com/culture/microsoft-disabling-word-2003s-...

Modern file systems address all of that and more - without the added complexity of a whole freaking RDBMS system! If you are going to change an application it makes far more sense to just update it to leverage the features of a modern file system.

Can't understand why this comment is so far down on the page. Apple Music has 98M users [1]. Let's say conservatively users listen to one song per day and update metadata on one song per month. That's 30 * 0.5 MB * 98M ~= 1 petabyte extra per month sent over the wire to save ~4 MB * 98M ~= 392 terabytes per month from being written to disk.

Those 0.5 MB presumably also need to be written to disk, making this a trading away of 392 TB in disk writes in exchange for 17 PB over the wire + 17 PB in disk writes? I'm sure this is overly simple and overlooks a lot of technical details, but I can't see how this can be worth it.

[1]: https://www.businessofapps.com/data/apple-music-statistics/

> In 2022, we shouldn't be reserving any area of a file as 'spare space' like this.

I completely understand your point, but maybe it would be a breaking change.

It wouldn't be. This spare space is entirely optional in the file format specification, and it's up to the tool that makes the file if it wants to include it or not.