News for hackers. Stuff that matters.
Re:Flattening ASTs (and Other Compiler Data Structures) by TazeTSchnitzel on Sunday July 02, 2023 @07:18AM (#36560372)
Blender (the 3D modelling software) is a fascinating case of this. To make loading and saving files fast and lossless, it uses identical on-disk and in-memory representations. In other words, everything is in an arena, and saving and loading is just a memcpy of the entire arena. Considering the massive potential complexity of a Blender project and the myriad issues you could have with serialisation and deserialisation, this seems like a great design to me.

The trade-off of course is that your data structure design is kinda sticky, insofar as you need to be able to open files for older versions.

Re:Flattening ASTs (and Other Compiler Data Structures) by glandium on Sunday July 02, 2023 @07:40AM (#36560483)
It also means you can't share files between little endians and big endians.
Re:Flattening ASTs (and Other Compiler Data Structures) by WJW on Sunday July 02, 2023 @07:56AM (#36560605)
Is this a real problem though? It seems to me that any system you would realistically want to run Blender on will be little-endian.
Re:Flattening ASTs (and Other Compiler Data Structures) by tjalfi on Sunday July 02, 2023 @03:47PM (#36564969)
It isn't a real problem. Blender supports only little-endian targets and the code byte swaps data files as needed. I guess someone could port it to AIX, z/OS, or Linux/zSeries, but that's unlikely at best.
Re:Flattening ASTs (and Other Compiler Data Structures) by rollcat on Sunday July 02, 2023 @08:05AM (#36560674)
There's almost certainly byte order+version markers in the header, a "happy path" that just memcpy's the structs, and a "safe path" that does byte order swapping, migrates the old structs to the current format, etc.

You can version the structs/loaders by keeping the relevant header files in their own version subfolders, and copious usage of sed. I've done this in a Django project to maintain support for ancient untouchable clients using an old version of the API, it's pretty manageable.

Re:Flattening ASTs (and Other Compiler Data Structures) by tjalfi on Sunday July 02, 2023 @04:08PM (#36565150)
I checked the repo. Blender detects files with a different endian and byte swaps if necessary.
Re:Flattening ASTs (and Other Compiler Data Structures) by TazeTSchnitzel on Monday July 03, 2023 @07:07AM (#36571293)
It doesn't, you just need to be careful to always read and write properties via accessor functions.
Re:Flattening ASTs (and Other Compiler Data Structures) by jeffreygoesto on Sunday July 02, 2023 @07:37AM (#36560469)
Wasn't that also the way Word used to dump structs onto disc in the "doc" days? I think I read that way also a major problem in writing converters, you baobab needed to decode Word's internal data structures which were of course undocumented.
Re:Flattening ASTs (and Other Compiler Data Structures) by orthoxerox on Sunday July 02, 2023 @09:18AM (#36561233)
I remember Spolsky saying Excel .xls worked like this, but I guess this should apply to .doc files as well.

This is usually orders of magnitude faster than serializing to/deserializing from a different storage format.

Re:Flattening ASTs (and Other Compiler Data Structures) by bestouff on Sunday July 02, 2023 @10:11AM (#36561644)
I don't think so. Serializing is IO-bound, if your format is "simple" (e.g. CBOR) you can't measure the difference.

All this look like BS to justify laziness to me, and makes loading/saving operations fragile and setting in-memory structures in stone.

Re:Flattening ASTs (and Other Compiler Data Structures) by Joker_vD on Sunday July 02, 2023 @10:25AM (#36561771)
> Serializing is IO-bound

Once upon a time, DMA did not meaningfully exist in the x86 world, so an IO-bound task was also CPU-bound.

Re:Flattening ASTs (and Other Compiler Data Structures) by bestouff on Sunday July 02, 2023 @12:56PM (#36563276)
Yes of course, but the original example was Blender, running on today's hardware. IMO this is just sloppy engineering in this case.
Re:Flattening ASTs (and Other Compiler Data Structures) by orthoxerox on Sunday July 02, 2023 @11:45AM (#36562551)
Don't forget that Excel file format is literally 30 years old. When you have 4MB of RAM in total for your OS, Excel and whatever part of the file you can fit into the rest and your CPU is an i486 if you're lucky, minimizing parsing starts to make lots of sense.
Re:Flattening ASTs (and Other Compiler Data Structures) by kragen on Tuesday July 04, 2023 @11:11PM (#36595132)
the cbor implementations i've had the misfortune to have to use are ridiculously slow, as slow as json parsers, perhaps because everything in cbor is variable-length

just because it's unreadable in a text editor doesn't mean it'll be fast

Re:Flattening ASTs (and Other Compiler Data Structures) by masklinn on Sunday July 02, 2023 @09:19AM (#36561240)
It's very common in old software and video games. Pre-97 office did that though with more formalisation than most (the binary format is now called Binary Interchange File Format, or BIFF, and is relatively generic, at the time MS almost certainly had utility libraries for working with BIFF).

This was a common source of portability issues for game saves.

Re:Flattening ASTs (and Other Compiler Data Structures) by jamesfinlayson on Sunday July 02, 2023 @10:07PM (#36567853)
Yep, pretty sure Valve's brief experiment with a 64-bit Source engine in late 2005 resulted in save games that didn't work with 32-bit Half-Life 2.
Re:Flattening ASTs (and Other Compiler Data Structures) by agumonkey on Sunday July 02, 2023 @05:32PM (#36565996)
And potentially photoshop.
Re:Flattening ASTs (and Other Compiler Data Structures) by seventhson on Sunday July 02, 2023 @08:37AM (#36560901)
I once worked on a very large commercial application that was based on a home-grown I/O framework which operated on similar principles.

It was a complete pain in the ass. You were constantly future-proofing your data structures because you knew you were going to be stuck with them for all eternity because the I/O framework was going to serialize them verbatim whether you liked it or not. Those were dark days...

Re:Flattening ASTs (and Other Compiler Data Structures) by Solvency on Sunday July 02, 2023 @10:37AM (#36561866)
I don't get it. Surely if you ever needed to, you could write an interpreter for loading tbe older files if you decided to abandon this 1:1 representation?
Re:Flattening ASTs (and Other Compiler Data Structures) by amomchilov on Sunday July 02, 2023 @11:12AM (#36562158)
I think the effort involved disincentivizes against it, and nudges people towards defensively adding spare fields or whatever.

The other comment pointed out that you can make a fall back migration code path that migrates over older file versions. That's the escape hatch if you have no other options

Re:Flattening ASTs (and Other Compiler Data Structures) by dragontamer on Sunday July 02, 2023 @01:08PM (#36563376)
You ever hear of .doc files?

https://learn.microsoft.com/en-us/openspecs/office_file_form...

Yeah. There is a reason XML was seen as the future back in the 90s.

Custom databases often get extended into custom filesystems. And these systems on top of systems get obscure features (like embedding excel sheets inside of Word) and... Thing get hairy.

Re:Flattening ASTs (and Other Compiler Data Structures) by 3cats-in-a-coat on Sunday July 02, 2023 @09:09AM (#36561153)
You can have the benefits of fast save and load without having the same representation. Stream changes to disk.
Re:Flattening ASTs (and Other Compiler Data Structures) by TazeTSchnitzel on Monday July 03, 2023 @07:08AM (#36571298)
Storing changes and the originals blew up badly for Microsoft Word in the late 1990's due to privacy concerns.
Re:Flattening ASTs (and Other Compiler Data Structures) by giovannibonetti on Sunday July 02, 2023 @09:16AM (#36561218)
> The trade-off of course is that your data structure design is kinda sticky, insofar as you need to be able to open files for older versions.

The missing piece is a way to evolve those data structures over time, in a similar way to database migrations. Only when loading data from an older version of the app those operations would need to be issued, and then the app could save the updated version in disk immediately to avoid incurring that cost again.

An example implementation of this concept is this project [1] that was created in the context or CRDTs. If it is not directly applicable, at least it should be a good inspiration.

[1] https://www.inkandswitch.com/cambria/

Re:Flattening ASTs (and Other Compiler Data Structures) by pavlov on Sunday July 02, 2023 @09:17AM (#36561227)
I believe Microsoft Word originally did this, and it was a major pain as the file format evolved.
Re:Flattening ASTs (and Other Compiler Data Structures) by Solvency on Sunday July 02, 2023 @10:35AM (#36561855)
So, the polar opposite of Photoshop's nightmare of a PSD format.
Re:Flattening ASTs (and Other Compiler Data Structures) by bjourne on Sunday July 02, 2023 @11:32AM (#36562400)
That is of course not Kosher at all. Compilers are generally free to layout and pad structs in whichever way they prefer, meaning that two compilers for the same platform might very well use incompatible layouts. It can lead to major problems if structs saved by say a 64-bit version of the program is to be loaded by the 32-bit version.
Re:Flattening ASTs (and Other Compiler Data Structures) by azornathogron on Sunday July 02, 2023 @01:26PM (#36563541)
Compilers (for C and C++) typically provide mechanisms to control padding and other layout requirements as necessary to make this work.

It's not that unusual in C and C++ code to define a struct that has a specific and well-defined memory layout. It's kosher, as long as you accept that you're working with a specific set of real compilers and use the appropriate #pragmas or other controls as needed to avoid undefined behaviour.

Re:Flattening ASTs (and Other Compiler Data Structures) by bjourne on Sunday July 02, 2023 @09:38PM (#36567685)
Yep, but then you force the compiler to choose a specific struct-layout which may be suboptimal if, for example, coding for simd. You also can't control the values of the struct padding bytes which may cause very interesting bugs.
Re:Flattening ASTs (and Other Compiler Data Structures) by grogenaut on Sunday July 02, 2023 @04:13PM (#36565191)
This is exactly how a PlayStation game I worked on worked and because everything was signed they were in no way worried about attackers. If you were able to alter the files on disk you had already owned your device. It meant loading was absurdly fast we pulled everything in the memory and then just patched pointer locations to where they actually got loaded