on linux opencv uses ffmpeg to videos iirc, i needed to install the gstreamer ffmpeg plugin (gst-libav on arch) and then it handled whatever format i threw at it (including mp4), although i used the python bindings if that makes any difference