What polymorphism is good at is building the black box, the soft boundaries, and that isn't a good thing to do for fine-grained details. I do occasionally find a use for extension, but it's at a scale closer to "call this entire subprogram as a kind of state machine", vs "write this algorithm as a collaboration of objects abstracted away from each other".
The "alternate hard and soft layers"[0] pattern comes to mind: if what you really need is modularity, going towards full dynamism and reflection seems like a better choice than to try to extend everything statically, which creates a very large latency issue(the default response to every form of change in a static system is "recompile from the beginning, precompute all answers"). At the same time, it's not a good fit to have a vast codebase do everything dynamically since the comprehensibility and throughput suffer.
Your idea of replacing polymorphic variants with sum types sounds interesting. It also strikes me as quite similar to compile time polymorphism (e.g. generics) where you are 1 to infinity in the design space but the generated program boils down to only the variants created. Language wise I really like the mix of sum types, traits and generics in Rust.
It seems the biggest thing OOP polymorphism has going for it is that it is "open", that is, the possible derived classes need not be known in advance. But, the fact that our toolchains use a separate link step, and that linkers remain primitive and link-time code generation is not widespread, is merely an historical accident. There's no a priori reason things have to be this way.
For example, an alternative would be if all of the source for an executable were in a single image file a la Smalltalk, and compilation consisted of converting this "database of code" into a single executable (no shared libraries!) in one big-bang compilation step (no object files, no static libraries!) You could even have something which syntactically resembled open polymorphism, maybe even using existing languages almost as-is, but instead of compiling down to type erasure and dynamic pointer casts, would compile down to Sum Types (variants). (The compiler could scan the image to find all of the derived types it could be and could construct a variant out of that.)
I'm not saying this is better, just putting it forth as an example of how things could be radically different than they are now (even for statically typed, compiled languages), had we made different choices. Consider the alternative, the situation we have now: we use a technique which could support anywhere from 1 to infinity polymorphic derived classes to implement interfaces which probably have approximately 2 different implementations in a typical finished program.
Perhaps an analogy would help: If we approached hardware the way we approach software, we would be using connectors which support between 1 and infinity pins everywhere when connectors which support N pins would suffice.