For what it's worth, I think that's true of the pre-1997 version of the format. For the format from 1997 to 2007, it's a proper format, albeit a bizarre one (looking over the spec makes me think the person tasked with making it was yanked off the filesystem team against their will)

Nowadays, it's a XML/ZIP thing.

With things like Flatbuffers, I sometimes feel we've regressed to these old formats that are just memory dumps.

A story I heard at Sun, which may be apocryphal but was fucking hilarious enough to be a repeatable rumor, was that a release of an early operating system in BETA was determined to be solid and tested and ready to release and ship to customers, so they simply changed the version string from something like "SunOS2.1BETA" to "SunOS2.1FCS" (First Customer Ship), and recompiled. But the change from a 12 character version to an 11 character version threw off the alignment of some important data structures somewhere in the kernel, and the entire OS ran MUCH SLOWER because of 68k unaligned memory accesses!

> With things like Flatbuffers, I sometimes feel we've regressed to these old formats

Sorta, but they're expressed in IDLs that are independent of a particular compiler. The downside is the drift between internal structs and the IDL structs, but the upside is you can use the same "memory dump" on interpreted runtimes or entirely different platforms (even those with different endians). Plus impls like protobuf help guard against breaking backward-compatibility by numbering fields and not "allowing" you to remove fields in ways that would change the structs' ABIs.

> With things like Flatbuffers, I sometimes feel we've regressed to these old formats that are just memory dumps.

It depends on what your application requirements are, but there are compelling arguments that on-disk / on-wire representations should match in-memory representations. It’s not too hard to end up with in a scenario where encoding / decoding times are a significant contribution to overall performance.

I worked at a company once where they, too, used raw memory dumps as their file format.

The compiler flags telling the compiler how to align structs were different for debug and non-debug builds. So, of course, the first thing I did, was to create non-debug build and try to open a file created by a debug-build-executable, the program crashed and burned without giving me a meaningful error message, it took me hours to understand my mistake.

Raw memory dumps are very neat efficiency-wise, but they are extremely fragile.

I'm curious, what was the reason for the difference? While the C language standard doesn't guarantee anything about alignment, I'm used to the behavior of compilers in practice being consistently 'align everything on its own size'; what changed between debug and release builds in that case?

In the non-debug build, the compiler was free to align structs as it saw fit, in the debug build, it was told to pack them really tight, no padding. Why? The only reason I can think of is to save a few precious bytes of disk space.

Primitives have not historically been aligned to their own size. There are plenty of older systems where double floats had 32-bit alignment instead of 64-bit, or 32-bit integers had 16-bit alignment.

They mentioned compiler flags, so my guess would be that they were doing something like tightly packing structs in the release build to lower memory usage or something similar.

Yes, I think that was the main reason.

It was kind if pointless, though, the biggest file I saw in my time there was around 10MB. How much bigger would that have been with padding? Even an overhead of 100% probably would not have had much of a performance impact and saving a few megabytes of disk space or RAM was not a very strong argument at the time any more (roughly 2007 - 2009)

On the other hand, this was a natural outgrowth of the other programmers' tendency to keep it simple and avoid bloat, which all in all I appreciated very much.

Or the opposite, as packing usually requires slower, unaligned accesses.

That would depend on how much that struct is created in memory vs the total code that uses it. If you have one instance in memory then yeah the code is probably bigger and slower (depending on arch). But if you have thousands of the structs in memory the speed and code size trade off may be worth it on a memory constrained system.

Debug also sometimes turns on overrun buffers so you can check for over/under runs in your code at debug time. Some compilers have this others dont.

FWIW here's the original 68k SimCity "Classic" (open source Micropolis) save function, which I've long since cleaned up and put in byte swapping to make it portable to SPARC and x86, but yes it is just writing out some big buffers of memory with a function that now swaps bytes:

https://github.com/SimHacker/micropolis/blob/master/Micropol...

Here's some original SimCity 2000 Mac code that saves the city into Mac resources (not the flat part of the file, but the Mac resource fork) -- CompGameWrite actually does some simple run length compression of the raw memory:

    Boolean DoSave(short VolNum,Byte *name){
    long count,CountTotal;
    short i,j,filnum;
    short x, y;
    Byte *SaveText[] = {"\pGame Saved As:",NIL,NIL};
    FInfo info;
    Byte *NotCity[] = {
        "\pSimCity 2000ª will not save over a non-city file.",
        "\pTry to save again using a different name.",
        NIL,
    };

        if (name[0]>20) name[0] = 20;

    #ifndef DEBUG
        if (GetFInfo(name,VolNum,&info)==0 && info.fdType!=CITYTYPE_ID) {
            MessageDialog(NotCity);
            return FALSE;
        }
    #endif

        if (FSOpen(name,VolNum,&filnum)!=noErr) {
            if (GameError(Create(name,VolNum,APPLICATION_ID,CITYTYPE_ID))) return FALSE;
            if (GameError(FSOpen(name,VolNum,&filnum))) return FALSE;
        }

        if (!WriteHeader(filnum,0L)) return FALSE;
        WriteLength = 4;        // includes 'SCDH' but not header

    //*** Write ***
        if (!MiscWrite(filnum)) return FALSE;
        if (!GameWrite(filnum,'ALTM',(Ptr)AltMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XTER',(Ptr)TerrainMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XBLD',(Ptr)BuildMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XZON',(Ptr)ZoneMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XUND',(Ptr)UnderMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XTXT',(Ptr)TextMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XLAB',(Ptr)LabelArray)) return FALSE;
        if (!CompGameWrite(filnum,'XMIC',(Ptr)MicroRecord)) return FALSE;
        if (!CompGameWrite(filnum,'XTHG',(Ptr)ThingList)) return FALSE;
        if (!CompGameWrite(filnum,'XBIT',(Ptr)BitsMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XTRF',(Ptr)TrafficMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XPLT',(Ptr)PolluteMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XVAL',(Ptr)ValueMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XCRM',(Ptr)CrimeMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XPLC',(Ptr)PoliceMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XFIR',(Ptr)FireMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XPOP',(Ptr)PopMap[0])) return FALSE;
        if (!CompGameWrite(filnum,'XROG',(Ptr)ROGMap[0])) return FALSE;

        if (!CompGameWrite(filnum,'XGRP',(Ptr)GraphData[0])) return FALSE;

    //*** Close ***
        WriteHeader(filnum,WriteLength);
        GameError(SetEOF(filnum,WriteLength+8));  // 8 bytes for file header
        GameError(FSClose(filnum));
        GameError(FlushVol(NIL,VolNum));

        BlockMove(name,CityStr,21);
        DateCashTitle();
        FilVolNum = VolNum;

    //*** SUCCESS! ***
        SaveText[1] = CityStr;
        MessageDialog(SaveText);
        return TRUE;
    }

And here's the Mac SimEarth save function, which looks like it just write out raw uncompressed memory -- the Mac memory manager has a GetPtrSize function that (obviously) tells you the size of the memory allocated for a pointer:

    DoSave(VolNum,name)
    int VolNum;
    Str255 name;
    {
    long count,CountTotal;
    int i,FilNum;

        if (EqualString(name,"\pSimEarth",FALSE,TRUE)) {
            MessageDialog( "\pERROR!!",
                            "\pThe name 'SimEarth' is reserved",
                            "\pfor this application.",1);
            return;
        }

        if (FSOpen(name,VolNum,&FilNum)) {
            if (GAIAError(Create(name,VolNum,'MYCR','SAVE'))) return;
            if (GAIAError(FSOpen(name,VolNum,&FilNum))) return;
        }
        else if (GAIAError(SetFPos(FilNum,fsFromStart,0L))) {
            GAIAError(FSClose(FilNum));
            return;
        }

    /**Write**/
        CountTotal = (count=GetPtrSize(Map[0]));
        if (GAIAError(FSWrite(FilNum,&count,Map[0]))) return;

        CountTotal += (count=GetPtrSize(Life[0]));
        if (GAIAError(FSWrite(FilNum,&count,Life[0]))) return;

        CountTotal += (count=GetPtrSize(OcTempMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,OcTempMap[0]))) return;

        CountTotal += (count=GetPtrSize(OcCurrentMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,OcCurrentMap[0]))) return;

        CountTotal += (count=GetPtrSize(DriftMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,DriftMap[0]))) return;

        CountTotal += (count=GetPtrSize(EventMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,EventMap[0]))) return;

        CountTotal += (count=GetPtrSize(SAirTempMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,SAirTempMap[0]))) return;

        CountTotal += (count=GetPtrSize(SCloudDensity[0]));
        if (GAIAError(FSWrite(FilNum,&count,SCloudDensity[0]))) return;

        CountTotal += (count=GetPtrSize(SAirCurrentMap[0]));
        if (GAIAError(FSWrite(FilNum,&count,SAirCurrentMap[0]))) return;

        PutParameters();
        CountTotal += (count=GetPtrSize(MiscHis));
        if (GAIAError(FSWrite(FilNum,&count,MiscHis))) return;

    /**Close**/
        GAIAError(SetEOF(FilNum,CountTotal));  
        GAIAError(FSClose(FilNum));
        GAIAError(FlushVol(NIL,VolNum));

        BlockMove(name,EwinStr,256); 
        SetWTitle(editWindow,EwinStr);
        MessageDialog("\pSave Complete:",EwinStr,"",1);
        EnableItem(GetMHandle(301),3);  /* Save */
        FilVolNum = VolNum;
    }