It's pretty fun to try writing a toy codec. It's not too hard if you don't have to worry about all the compatibility and generality stuff. Try to make something that outperforms the standard but only on a specific video. :)

Also I learned a lot by messing with the ffmpeg source, it's pretty readable.