Every time you think you need a custom file format for a given piece of software... you most likely don't. Just use SQLite. You can use the standard OS file Open/Save dialog box and users will never know the difference.
Yes, that means you now have to write SQL statements to manipulate your data. But that also means that you can get lots of complicated data structures on disk and can manipulate them easily, even outside your own software - just fire up the sqlite CLI and point to your file. There are GUIs as well.
Things like UNDO/REDO can also be had almost trivially (see https://www.sqlite.org/undoredo.html)
Sometimes this also means you don't need an external RDBMS even for web apps. I've seen so many apps which co-locate a small database in the same box that might as well have been a single sqlite file. I'm actually maintaining one right now that, although relatively important, will only ever be a single box. But sqlite wasn't 'enterprisey' enough, had to use PG. For a couple of tables.
https://www.sqlite.org/whentouse.html
Subscribe for my next rant on another underused piece of tech... Lua :)
For on-disk document storage I think Apple mostly uses a mix of plain folders with magic extensions (“packages”) and ZIP files nowadays, although there are definitely a lot of exceptions. SQLite isn’t that great for binary blob storage (relatively speaking) so a folder structure is still more useful there, IMO.
Consortia have been tried many times, but few as successful as the SQLite Consortium.
What a great model. Open source for the main thing, proprietary test suite.
This model does require building something everyone needs, years of patient care and feeding to make it wildly popular, and a fantastic test suite. So it's not exactly easy to pull off. But it is brilliant, if you can do it.
[0] Even just contributing is very difficult. Try to contribute anything other than a trivial bug fix, and you'll find that a) unless it's utterly trivial, the devs are not interested, b) they are dead serious about checking that the contributor has their employer's permission to place that contribution in the public domain.
Not only can you get a SQLite database (which can be used as a application file format, suitable for many uses), but there are functions such as sqlite3_mprintf() and sqlite3_str_new() and stuff like that; and furthermore, it can allow the user to query it too with SQL commands, even within the program, and to allow user customizability by the use of SQL codes, too, and it allows the user to deal with the file even without your software in some cases (not needing to write a separate decoder). Sometimes virtual tables are useful for doing some things. And, this is in addition to being able to use SQLite as a file format.