I guess you can kind of mod Ableton Live using Max4Live, although while you can add new tools and connect to existing tools, you can't (last time I played with M4L a few years ago) change any of the core widgets, so its definitely not as deep as what I understand you to be saying.
If you look how much time gets invested into modding some game's UI, how much neat things could be done, if the whole UI of a DAW is easily moddable.