I also do appreciate MP3's simplicity, in the sense that it's just a series of concatenated frames. It makes it really easy to just fling them over the network and get streaming audio working on a client. And there's also somewhat of an elegance (a very ugly and hacky elegance, mind you) to being able to exploit decoders ignoring malformed frames.
For example if you open an MP3 IceCast stream via HTTP in a media player like VLC, the server (if it realizes, via the HTTP request headers, that you're Icecast-aware) will occasionally barf the name of the current and following song into the MP3 stream. Meaning you don't need any higher-level streaming protocol to deal with, and can just send raw frames over the wire, where the MP3 decoder will ignore the song title as a malformed MP3 frame, but VLC will pick-up on the song title and display it for you as the server cycles from one song to the next. Kinda handy, because VLC will make use of this metadata, but at the same time, the MP3 stream's URL will also work in a web browser too, since the browser won't need to know how to deal with a higher-level protocol before being able to start receiving those frames.
Actually, nevermind all that. MP3 bad.
Oh and by the way. For this exact purpose, ID3v2 has a flag called "unsynchronized" that means all 0xFF bytes in that field are replaced with 0xFF00, so that players won't play the wrong thing. You're supposed to replace them back when parsing the tag. A very hacky solution.
This is somewhat ironic given the fact that quite large part of MPEG specification deals with various framing and metadata structures (on the other hand the overall architecture of all that is best described as "overengineered", so ignoring it makes some kind of sense).