I'm aware of the raytracer library example. The solution to that is to not use types to represent shapes. Instead, classify shapes by primitive types (triangles, quads, bezier curves, NURBS, etc.) and use a closed sum for those. It's far less common for a user to want to create a new primitive type and you can always use an escape hatch in the closed sum that allows the user to define their own primitive along with a function to draw it in terms of one of the other primitives.
For what it's worth, I do think people underestimate the applicability of closed sum types.