News for hackers. Stuff that matters.
Re:The 0.5 MB of nothing in all Apple Music files (2020) by dotancohen on Thursday April 07, 2022 @08:49AM (#30943729)

  > In 2022, we shouldn't be reserving any area of a file as 'spare space' like this.
In 2022, we should have file formats that are not naive of the underlying filesystem. I store my text notes in Git, but I also have two LibreOffice documents in there because org-mode doesn't handle their contents well. It would be nice of only 5% of the file had to be rewritten when I change a single word, rather than the current 91% of the file. A 5% hit to storage for a 91% reduction each time I update it would be a great trade off.
Re:The 0.5 MB of nothing in all Apple Music files (2020) by coder543 on Thursday April 07, 2022 @09:32AM (#30944183)
Under the hood, odt and docx files are zip files.[0] The “91%” change has nothing to do with rewriting the file; it mostly has to do with zip files and how poorly git handles binary diffing. Plaintext files are rewritten on disk every time you save them in virtually every editor and that doesn’t cause problems for git.

I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again.

Alternatively, you could just switch to a text-based format like rtf, as long as you don’t need any specific features from odt or docx.

(EDIT: someone else mentions a promising sounding “flat xml” format, which I’ve never encountered before.)

[0]: https://superuser.com/a/1356829

Re:The 0.5 MB of nothing in all Apple Music files (2020) by 0xffff2 on Thursday April 07, 2022 @03:49PM (#30949540)
>I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again.

I have a current project where one of the other developers has tried this. It might work for a solo project, but with multiple developers on Windows (Excel) and Linux (Open Office), I find that Open Office feels a need to continually fuck with unrelated fields. Editing a single field in an xslx file results in changes in a dozen files in the unpacked version. Making any more complicated changes results in a diff that's impossible to manage.

Re:The 0.5 MB of nothing in all Apple Music files (2020) by coder543 on Thursday April 07, 2022 @03:54PM (#30949590)
Do you actually mean OpenOffice, or do you mean LibreOffice? I would hope everyone is using the much more improved LibreOffice by now instead of OpenOffice.

Still, interesting to hear some feedback on how well (or not) that idea has worked for some.

The Flat XML option seems to be the best approach if you need to edit documents more complex than what can be represented in either markdown or rtf (rich text format).

Re:The 0.5 MB of nothing in all Apple Music files (2020) by sjhuang26 on Thursday April 07, 2022 @09:32AM (#30944192)
In your case, LibreOffice allows you to save the file as Flat XML (e.g., .fodt), and the document will be written as a single uncompressed file. Unfortunately, few apps have a feature like this.
Re:The 0.5 MB of nothing in all Apple Music files (2020) by dotancohen on Thursday April 07, 2022 @09:47AM (#30944369)
This is perfect for my use case, thank you! The .fodt file is 150K, vs 117K for the .odt file. But after changing a word less than 20 of the 1983 lines in the file have changed.

  $ ls -hl todo.fodt
  -rw-rw-r-- 1 dotancohen dotancohen 150K Apr  7 16:41 todo.fodt
  $ ls -hl todo.odt
  -rwxrwxr-x 1 dotancohen dotancohen 117K Apr  6 12:01 todo.odt
  $ wc -l todo.fodt
  1983 todo.fodt
  $ cp todo.fodt todo-edit-one-line.fodt
  $ libreoffice todo-edit-one-line.fodt
  $ diff todo.fodt todo-2.fodt | grep "+ " | wc -l
  19
Re:The 0.5 MB of nothing in all Apple Music files (2020) by ktpsns on Thursday April 07, 2022 @09:39AM (#30944282)
Wow, thanks for the Flat OpenDocument file type! Such kind of files are much better to treat with git (i.e. human readable diffs, for instance). However, unfortunately, these files tend to get very large (200kB ODS vs. 7MB FODS). What a pity the XML itself is not less verbose.
Re:The 0.5 MB of nothing in all Apple Music files (2020) by lelanthran on Thursday April 07, 2022 @09:47AM (#30944364)
I dunno if that will help. I know Dia lets you store the uncompressed XML, but I have seen it sometimes have a 100% diff due to the serialiser moving *everything* around when all that happened was that I rearranged half the blocks on the diagram.
Re:The 0.5 MB of nothing in all Apple Music files (2020) by nyanpasu64 on Thursday April 07, 2022 @09:08AM (#30943933)
One option is to store files as a database like SQLite, which I assume is designed around not rewriting the whole file on every change. For example Audacity 3.x does this.

However SQLite stores multiple files on disk for crash persistence, complicating matters. But then again so does Audacity 2, and Microsoft Office to an extent (file locking rather than data integrity).

Microsoft Word used to only append changes to the document on save rather than rewriting the whole file, but this was disabled since if you deleted text and saved the file, it remained in the document. https://www.cnet.com/culture/microsoft-disabling-word-2003s-...

Re:The 0.5 MB of nothing in all Apple Music files (2020) by EricE on Thursday April 07, 2022 @09:25AM (#30944111)
Modern file systems address all of that and more - without the added complexity of a whole freaking RDBMS system! If you are going to change an application it makes far more sense to just update it to leverage the features of a modern file system.