Have you tried using Julia? Tom Breloff's Plots.jl[0] package has a built-in @gif macro, which already relies on ImageMagick and/or FFmpeg (depending on which one of them is present). So it shouldn't be too hard to extend that to mp4 (just pass different output parameters to FFmpeg). In fact, he has an issue open for it[1] so he appears to be working on that.

[0] https://juliaplots.github.io/#simple-is-beautiful

[1] https://github.com/tbreloff/Plots.jl/issues/320