I mean, if you allow every compositor to implement its own plugin system, then you end up with a lot of fragmentation and you effectively lock the user into one particular compositor.

So then you have to come up with a plugin system which everyone wants, which is probably not even possible because the definition of "plugin" is so vague that it could mean anything. If you ask me, fragmentation is just the default state of things in open source.

Yes, the default state is the state you get if you don't put any work into it. It is the "easy" state. Defining a good plugin system is not easy, true.

I remember at the time the Compiz developers were trying to get everyone else to either base their compositor on Compiz, or build in support for Compiz plugins. That might have been feasible if there was a working design, but Compiz plugins were very hacky and the code was a mess. IIRC the only desktop that actually did it was Unity, and that was discontinued. So you can see some of the troubles with that approach.