I'd say do both, actually. But versioning is more important IMO because it gives you the freedom to potentially start from scratch keeping only a small portion of the header.

On mainstream file formats... It's a mixed bag. Image formats — JPEG and PNG especially — are extensible and reasonably easy to parse. It's fairly trivial to get the image dimensions out of one of these without decoding the compressed data. I did as well write a JPEG decoder out of curiosity once to understand the compression algorithm better — it's an interesting exercise, really, every software developer should try it at some point.

But the worst format I've ever worked with is MP3. It's an absolute mess. First, there are two kinds of ID3 tags. These store metadata that your player displays. ID3v1 is a fixed-length, fixed-layout thing that goes on the end of the file. ID3v2 is an extensible, I'd say way too extensible, chunked thing capable of storing literally anything, including jpegs of cover arts, that goes on the beginning of the file. But none of them store the duration of the file. You're supposed to chop the tags off the ends of it, then find the first frame of encoded data by searching for the pattern 0xFFFx, read its header, and determine the byte length, bitrate, the sampling rate and ultimately the duration of a that frame using several lookup tables. Now that you know how much audio each frame contains, and how long it is, you take the size of the file (minus tags of course) and divide it by the frame length, then multiply by the frame duration. That's how you get the duration of an MP3. A constant-bitrate one. And to seek within an MP3, you calculate the offset into the file and round it to the nearest frame size and just start playing from there. It gets even worse with VBR, because now you can no longer rely on frames being the same byte length, but I don't really remember the details any more. The gist of it is that there's "header" encoded into the very first frame in the file, and there are two kinds of these headers, and there's a sort of lookup table in it, among other things, to help you seek into the right part of the file because the byte offsets don't linearly correspond to the playback time in a VBR file. After you seek, you have to go back and forth to find the 0xFFFx and play from there. Or not, because sometimes there's a 0xFFFx in the middle of a frame too, so you have to have some heuristics to detect that it's the real one.

That is because MP3 is not a file format in the first place. The file is originally simply an stream of MP3 frames written into a file without any structure and usually with technically invalid and undecodable frame at both ends (due to how MP3 compression works). Because it is originally designed for transmission across some kind of somewhat unreliable network, there are resynchronization structures in the stream and decoders tend to be able to ignore various kinds of totally invalid crap in the input data, this feature is exploited by all the ID3vX formats to essentially embed arbitrary data into the file. Several other commonly used "MPEG something" "file formats" are exactly the same thing.

This is somewhat ironic given the fact that quite large part of MPEG specification deals with various framing and metadata structures (on the other hand the overall architecture of all that is best described as "overengineered", so ignoring it makes some kind of sense).

This pretty much nails it.

I also do appreciate MP3's simplicity, in the sense that it's just a series of concatenated frames. It makes it really easy to just fling them over the network and get streaming audio working on a client. And there's also somewhat of an elegance (a very ugly and hacky elegance, mind you) to being able to exploit decoders ignoring malformed frames.

For example if you open an MP3 IceCast stream via HTTP in a media player like VLC, the server (if it realizes, via the HTTP request headers, that you're Icecast-aware) will occasionally barf the name of the current and following song into the MP3 stream. Meaning you don't need any higher-level streaming protocol to deal with, and can just send raw frames over the wire, where the MP3 decoder will ignore the song title as a malformed MP3 frame, but VLC will pick-up on the song title and display it for you as the server cycles from one song to the next. Kinda handy, because VLC will make use of this metadata, but at the same time, the MP3 stream's URL will also work in a web browser too, since the browser won't need to know how to deal with a higher-level protocol before being able to start receiving those frames.

Actually, nevermind all that. MP3 bad.

IIRC IceCast essentially extends/abuses HTTP/1.1 chunked encoding to frame such metadata. The original Nullsoft's ShoutCast protocol simply multiplexes fixed size blocks of mp3 data with length prefixed metadata packets. In both cases you can pretty easily strip the metadata out.

Back when di.fm was an actual online radio and gave you a stream URL to play in a player of your choice, it was dead simple. I don't know what protocol it used, but there was no chunked encoding and no content-length, simply a 200 OK HTTP response and then an endless MP3 "file" with ID3 tags inserted every now and then. You could play it with Android's MediaPlayer in like 3 lines of code.

Oh and by the way. For this exact purpose, ID3v2 has a flag called "unsynchronized" that means all 0xFF bytes in that field are replaced with 0xFF00, so that players won't play the wrong thing. You're supposed to replace them back when parsing the tag. A very hacky solution.

You should have a version, but given the (admittedly unrealistic) choice between versioning and extensibility I’ll take extensibility every time. There are plenty of formats where there’s a version tag and it’s never been bumped past “1”.

Early compressed audio / video formats were generally a total mess, with a couple exceptions like MOV, so I’m not surprised that MP3 is horrible.

I second your observations on JPEG. It turns out that you can use the same code to skim through jpeg, lossless jpeg, Jpeg2000 and jpeg-ls headers. Impressive in it’s simplicity.

Every JPEG 2000 encoder I've seen encapsulates the JP2K data in an MOV/MP4-like format (or if one likes using many syllables, technically the ISO Base Media File Format). Even when using Motion JPEG 2000 image sequences embedded as frames in an MP4 video file, each separate JP2K frame includes the MP4-like container as you would see if the stand-alone JP2K file was sitting on-disk.

Anyway, it is indeed cool that you can encode JPEG-LS, Lossless JPEG and friends in the exact same format as you see baseline JPEG images in - but just in different application segments! (eg, baseline=SOF0/SOF1 segment, Lossless JPEG=SOF3, JPEG-LS=SOF48, etc.).