> Depending on prebuilt C++ libraries hampers flexibility (eg. you can't customize the memory management or rasterization pipeline of matplotlib).

But what is the counterfactual? Implementing the whole thing in Python? It seems much more work than forking/fixing matplotlib.