That's why you version your file formats and network protocols. Actually, designing file formats and network protocols is one of the few areas in software engineering where you do really need future-proof design and extreme extensibility, because once you release the thing, these are set in stone. Yet not many people seem to realize this. They instead "future-proof" their code with useless abstraction layers.

Anyway. Versioning helps you avoid ugly workarounds if you need to extend your format in the future in ways that its current version doesn't allow. You then keep the code for older versions as a backwards-compatibility-only kind of deal and move on to the new one.

Versioning is one approach, but I favor making the format extensible in the first place. For example, if you pick an XML format you can add new attributes and tags, if you pick a Protobuf format you can add new fields. “Extensible” sounds like it can be a real mess but there are effective strategies to minimize the mess.

There are also various chunked formats like AIFC (AIFF) and PNG which can be extended by defining new chunk types, without needing to change versioning. AIFC includes the APPL/stoc chunk and PNG has various mechanisms.

There are a few problems with versioning file formats. One is that you often end up with 'if (version > 3)' scattered across your code base or other nonsense. Another problem is that it is easy to accidentally mark the wrong version, either a version which is too low (because you wrote something to the file and forgot to make your encoder bump the version properly) or a version which is too high (because you didn’t bother to use the minimum version your data requires).

I'd say do both, actually. But versioning is more important IMO because it gives you the freedom to potentially start from scratch keeping only a small portion of the header.

On mainstream file formats... It's a mixed bag. Image formats — JPEG and PNG especially — are extensible and reasonably easy to parse. It's fairly trivial to get the image dimensions out of one of these without decoding the compressed data. I did as well write a JPEG decoder out of curiosity once to understand the compression algorithm better — it's an interesting exercise, really, every software developer should try it at some point.

But the worst format I've ever worked with is MP3. It's an absolute mess. First, there are two kinds of ID3 tags. These store metadata that your player displays. ID3v1 is a fixed-length, fixed-layout thing that goes on the end of the file. ID3v2 is an extensible, I'd say way too extensible, chunked thing capable of storing literally anything, including jpegs of cover arts, that goes on the beginning of the file. But none of them store the duration of the file. You're supposed to chop the tags off the ends of it, then find the first frame of encoded data by searching for the pattern 0xFFFx, read its header, and determine the byte length, bitrate, the sampling rate and ultimately the duration of a that frame using several lookup tables. Now that you know how much audio each frame contains, and how long it is, you take the size of the file (minus tags of course) and divide it by the frame length, then multiply by the frame duration. That's how you get the duration of an MP3. A constant-bitrate one. And to seek within an MP3, you calculate the offset into the file and round it to the nearest frame size and just start playing from there. It gets even worse with VBR, because now you can no longer rely on frames being the same byte length, but I don't really remember the details any more. The gist of it is that there's "header" encoded into the very first frame in the file, and there are two kinds of these headers, and there's a sort of lookup table in it, among other things, to help you seek into the right part of the file because the byte offsets don't linearly correspond to the playback time in a VBR file. After you seek, you have to go back and forth to find the 0xFFFx and play from there. Or not, because sometimes there's a 0xFFFx in the middle of a frame too, so you have to have some heuristics to detect that it's the real one.

That is because MP3 is not a file format in the first place. The file is originally simply an stream of MP3 frames written into a file without any structure and usually with technically invalid and undecodable frame at both ends (due to how MP3 compression works). Because it is originally designed for transmission across some kind of somewhat unreliable network, there are resynchronization structures in the stream and decoders tend to be able to ignore various kinds of totally invalid crap in the input data, this feature is exploited by all the ID3vX formats to essentially embed arbitrary data into the file. Several other commonly used "MPEG something" "file formats" are exactly the same thing.

This is somewhat ironic given the fact that quite large part of MPEG specification deals with various framing and metadata structures (on the other hand the overall architecture of all that is best described as "overengineered", so ignoring it makes some kind of sense).

This pretty much nails it.

I also do appreciate MP3's simplicity, in the sense that it's just a series of concatenated frames. It makes it really easy to just fling them over the network and get streaming audio working on a client. And there's also somewhat of an elegance (a very ugly and hacky elegance, mind you) to being able to exploit decoders ignoring malformed frames.

For example if you open an MP3 IceCast stream via HTTP in a media player like VLC, the server (if it realizes, via the HTTP request headers, that you're Icecast-aware) will occasionally barf the name of the current and following song into the MP3 stream. Meaning you don't need any higher-level streaming protocol to deal with, and can just send raw frames over the wire, where the MP3 decoder will ignore the song title as a malformed MP3 frame, but VLC will pick-up on the song title and display it for you as the server cycles from one song to the next. Kinda handy, because VLC will make use of this metadata, but at the same time, the MP3 stream's URL will also work in a web browser too, since the browser won't need to know how to deal with a higher-level protocol before being able to start receiving those frames.

Actually, nevermind all that. MP3 bad.

IIRC IceCast essentially extends/abuses HTTP/1.1 chunked encoding to frame such metadata. The original Nullsoft's ShoutCast protocol simply multiplexes fixed size blocks of mp3 data with length prefixed metadata packets. In both cases you can pretty easily strip the metadata out.

Back when di.fm was an actual online radio and gave you a stream URL to play in a player of your choice, it was dead simple. I don't know what protocol it used, but there was no chunked encoding and no content-length, simply a 200 OK HTTP response and then an endless MP3 "file" with ID3 tags inserted every now and then. You could play it with Android's MediaPlayer in like 3 lines of code.

Oh and by the way. For this exact purpose, ID3v2 has a flag called "unsynchronized" that means all 0xFF bytes in that field are replaced with 0xFF00, so that players won't play the wrong thing. You're supposed to replace them back when parsing the tag. A very hacky solution.

You should have a version, but given the (admittedly unrealistic) choice between versioning and extensibility I’ll take extensibility every time. There are plenty of formats where there’s a version tag and it’s never been bumped past “1”.

Early compressed audio / video formats were generally a total mess, with a couple exceptions like MOV, so I’m not surprised that MP3 is horrible.

I second your observations on JPEG. It turns out that you can use the same code to skim through jpeg, lossless jpeg, Jpeg2000 and jpeg-ls headers. Impressive in it’s simplicity.

Every JPEG 2000 encoder I've seen encapsulates the JP2K data in an MOV/MP4-like format (or if one likes using many syllables, technically the ISO Base Media File Format). Even when using Motion JPEG 2000 image sequences embedded as frames in an MP4 video file, each separate JP2K frame includes the MP4-like container as you would see if the stand-alone JP2K file was sitting on-disk.

Anyway, it is indeed cool that you can encode JPEG-LS, Lossless JPEG and friends in the exact same format as you see baseline JPEG images in - but just in different application segments! (eg, baseline=SOF0/SOF1 segment, Lossless JPEG=SOF3, JPEG-LS=SOF48, etc.).

Data structures and object protocols are the analog to file formats and network protocols. They're not immutable but unless you want software changes to affect everything, you want something at least a bit more stable

Note that the issue here is less the part that you quoted and more the two sentences surrounding it, PSD has all of

* unaligned chunks

* aligned chunks with alignment included in size

* aligned chunks with alignment not included in size

The problem is not the specific choice, "Either one of these three behaviours would be fine", it's that "PSD, of course, uses all three, and more."

As I understand it that's more or less what the older MS Office formats were, just a dump of the entire OLE object representing the document as-is.

I believe SimCity used a similar strategy as well, the .cty and I think SC2k's .s2k formats just dumped the in-memory representation to disk,.

For what it's worth, I think that's true of the pre-1997 version of the format. For the format from 1997 to 2007, it's a proper format, albeit a bizarre one (looking over the spec makes me think the person tasked with making it was yanked off the filesystem team against their will)

Nowadays, it's a XML/ZIP thing.

With things like Flatbuffers, I sometimes feel we've regressed to these old formats that are just memory dumps.

A story I heard at Sun, which may be apocryphal but was fucking hilarious enough to be a repeatable rumor, was that a release of an early operating system in BETA was determined to be solid and tested and ready to release and ship to customers, so they simply changed the version string from something like "SunOS2.1BETA" to "SunOS2.1FCS" (First Customer Ship), and recompiled. But the change from a 12 character version to an 11 character version threw off the alignment of some important data structures somewhere in the kernel, and the entire OS ran MUCH SLOWER because of 68k unaligned memory accesses!

> With things like Flatbuffers, I sometimes feel we've regressed to these old formats

Sorta, but they're expressed in IDLs that are independent of a particular compiler. The downside is the drift between internal structs and the IDL structs, but the upside is you can use the same "memory dump" on interpreted runtimes or entirely different platforms (even those with different endians). Plus impls like protobuf help guard against breaking backward-compatibility by numbering fields and not "allowing" you to remove fields in ways that would change the structs' ABIs.

> With things like Flatbuffers, I sometimes feel we've regressed to these old formats that are just memory dumps.

It depends on what your application requirements are, but there are compelling arguments that on-disk / on-wire representations should match in-memory representations. It’s not too hard to end up with in a scenario where encoding / decoding times are a significant contribution to overall performance.

I worked at a company once where they, too, used raw memory dumps as their file format.

The compiler flags telling the compiler how to align structs were different for debug and non-debug builds. So, of course, the first thing I did, was to create non-debug build and try to open a file created by a debug-build-executable, the program crashed and burned without giving me a meaningful error message, it took me hours to understand my mistake.

Raw memory dumps are very neat efficiency-wise, but they are extremely fragile.

I'm curious, what was the reason for the difference? While the C language standard doesn't guarantee anything about alignment, I'm used to the behavior of compilers in practice being consistently 'align everything on its own size'; what changed between debug and release builds in that case?

In the non-debug build, the compiler was free to align structs as it saw fit, in the debug build, it was told to pack them really tight, no padding. Why? The only reason I can think of is to save a few precious bytes of disk space.

Primitives have not historically been aligned to their own size. There are plenty of older systems where double floats had 32-bit alignment instead of 64-bit, or 32-bit integers had 16-bit alignment.

They mentioned compiler flags, so my guess would be that they were doing something like tightly packing structs in the release build to lower memory usage or something similar.

Yes, I think that was the main reason.

It was kind if pointless, though, the biggest file I saw in my time there was around 10MB. How much bigger would that have been with padding? Even an overhead of 100% probably would not have had much of a performance impact and saving a few megabytes of disk space or RAM was not a very strong argument at the time any more (roughly 2007 - 2009)

On the other hand, this was a natural outgrowth of the other programmers' tendency to keep it simple and avoid bloat, which all in all I appreciated very much.

Or the opposite, as packing usually requires slower, unaligned accesses.

That would depend on how much that struct is created in memory vs the total code that uses it. If you have one instance in memory then yeah the code is probably bigger and slower (depending on arch). But if you have thousands of the structs in memory the speed and code size trade off may be worth it on a memory constrained system.

Debug also sometimes turns on overrun buffers so you can check for over/under runs in your code at debug time. Some compilers have this others dont.

FWIW here's the original 68k SimCity "Classic" (open source Micropolis) save function, which I've long since cleaned up and put in byte swapping to make it portable to SPARC and x86, but yes it is just writing out some big buffers of memory with a function that now swaps bytes:

https://github.com/SimHacker/micropolis/blob/master/Micropol...

Here's some original SimCity 2000 Mac code that saves the city into Mac resources (not the flat part of the file, but the Mac resource fork) -- CompGameWrite actually does some simple run length compression of the raw memory:

    Boolean DoSave(short VolNum,Byte *name){
    long count,CountTotal;
    short i,j,filnum;
    short x, y;
    Byte *SaveText[] = {"\pGame Saved As:",NIL,NIL};
    FInfo info;
    Byte *NotCity[] = {
        "\pSimCity 2000ª will not save over a non-city file.",
        "\pTry to save again using a different name.",
        NIL,
    };

        if (name[0]>20) name[0] = 20;

    #ifndef DEBUG
        if (GetFInfo(name,VolNum,&info)==0 && info.fdType!=CITYTYPE_ID) {
            MessageDialog(NotCity);
            return FALSE;
        }
    #endif

        if (FSOpen(name,VolNum,&filnum)!=noErr) {
            if (GameError(Create(name,VolNum,APPLICATION_ID,CITYTYPE_ID))) return FALSE;
            if (GameError(FSOpen(name,VolNum,&filnum))) return FALSE;
        }

        if (!WriteHeader(filnum,0L)) return FALSE;
        WriteLength = 4;        // includes 'SCDH' but not header

    //*** Write ***
        if (!MiscWrite(filnum)) return FALSE;
        if (!GameWrite(filnum,'ALTM',(Ptr)AltMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XTER',(Ptr)TerrainMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XBLD',(Ptr)BuildMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XZON',(Ptr)ZoneMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XUND',(Ptr)UnderMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XTXT',(Ptr)TextMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XLAB',(Ptr)LabelArray)) return FALSE;
        if (!CompGameWrite(filnum,'XMIC',(Ptr)MicroRecord)) return FALSE;
        if (!CompGameWrite(filnum,'XTHG',(Ptr)ThingList)) return FALSE;
        if (!CompGameWrite(filnum,'XBIT',(Ptr)BitsMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XTRF',(Ptr)TrafficMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XPLT',(Ptr)PolluteMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XVAL',(Ptr)ValueMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XCRM',(Ptr)CrimeMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XPLC',(Ptr)PoliceMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XFIR',(Ptr)FireMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XPOP',(Ptr)PopMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XROG',(Ptr)ROGMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XGRP',(Ptr)GraphData[0])) return FALSE;

    //*** Close ***
        WriteHeader(filnum,WriteLength);
        GameError(SetEOF(filnum,WriteLength+8));  // 8 bytes for file header
        GameError(FSClose(filnum));
        GameError(FlushVol(NIL,VolNum));

        BlockMove(name,CityStr,21);
        DateCashTitle();
        FilVolNum = VolNum;

    //*** SUCCESS! ***
        SaveText[1] = CityStr;
        MessageDialog(SaveText);
        return TRUE;
    }

And here's the Mac SimEarth save function, which looks like it just write out raw uncompressed memory -- the Mac memory manager has a GetPtrSize function that (obviously) tells you the size of the memory allocated for a pointer:

    DoSave(VolNum,name)
    int VolNum;
    Str255 name;
    {
    long count,CountTotal;
    int i,FilNum;

        if (EqualString(name,"\pSimEarth",FALSE,TRUE)) {
            MessageDialog( "\pERROR!!",
                            "\pThe name 'SimEarth' is reserved",
                            "\pfor this application.",1);
            return;
        }

        if (FSOpen(name,VolNum,&FilNum)) {
            if (GAIAError(Create(name,VolNum,'MYCR','SAVE'))) return;
            if (GAIAError(FSOpen(name,VolNum,&FilNum))) return;
        }
        else if (GAIAError(SetFPos(FilNum,fsFromStart,0L))) {
            GAIAError(FSClose(FilNum));
            return;
        }

    /**Write**/
        CountTotal = (count=GetPtrSize(Map[0]));
        if (GAIAError(FSWrite(FilNum,&count,Map[0]))) return;

        CountTotal += (count=GetPtrSize(Life[0]));
        if (GAIAError(FSWrite(FilNum,&count,Life[0]))) return;

        CountTotal += (count=GetPtrSize(OcTempMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,OcTempMap[0]))) return;

        CountTotal += (count=GetPtrSize(OcCurrentMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,OcCurrentMap[0]))) return;

        CountTotal += (count=GetPtrSize(DriftMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,DriftMap[0]))) return;

        CountTotal += (count=GetPtrSize(EventMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,EventMap[0]))) return;

        CountTotal += (count=GetPtrSize(SAirTempMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,SAirTempMap[0]))) return;

        CountTotal += (count=GetPtrSize(SCloudDensity[0]));
        if (GAIAError(FSWrite(FilNum,&count,SCloudDensity[0]))) return;

        CountTotal += (count=GetPtrSize(SAirCurrentMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,SAirCurrentMap[0]))) return;

        PutParameters();
        CountTotal += (count=GetPtrSize(MiscHis));
        if (GAIAError(FSWrite(FilNum,&count,MiscHis))) return;

    /**Close**/
        GAIAError(SetEOF(FilNum,CountTotal));  
        GAIAError(FSClose(FilNum));
        GAIAError(FlushVol(NIL,VolNum));

        BlockMove(name,EwinStr,256); 
        SetWTitle(editWindow,EwinStr);
        MessageDialog("\pSave Complete:",EwinStr,"",1);
        EnableItem(GetMHandle(301),3);  /* Save */
        FilVolNum = VolNum;
    }

That was very common not long ago. Even the JPEG/EXIF image file format is designed that way. So it's efficient for reading and writing, but introduces a lot of potential bugs with alignment and chunk size issues. Inserting an additional EXIF tag is a huge hassle because then you have to recalculate all the pointers, even those in other data chunks.

One form of true insanity is probably JPEG-encoded TIFF files. Apparently it's so bad that for the next version of the TIFF specification they are taking that insane approach out completely.

I can't do justice to the article on TIFF's problems with JPEG (http://www.simplesystems.org/libtiff/TIFFTechNote2.html), but my understanding is that:

1) Some JPEG-specific data is moved outside of the actual JPEG bitstream tag, and into separate TIFF tags, making editing JPEG-in-TIFF files non-trivial. 2) Size is not encoded in some of these fields, so the TIFF editor you're writing will have to partially implement a JPEG decoder just to know the size of some of those TIFF tags, and 3) Some tags/fields are pointers into other parts of the TIFF file, meaning if you edit the file, you'll have to update the file in many places.

(As a quick aside on insane formats, may I also mention the EPWING dictionary format?)

In practice, it's the simplest and most efficient way, since the majority of the time you're not going to be dealing with any insanely weird architectures and stuff like 9-bit bytes has thankfully disappeared from common use.

I do wish they'd pack their structures, however.