They were an ungodly mess, in my view, compared to modern approaches which is what matters in the context of this discussion. The issue also wasn't XUL but everything else related to how extensions functioned (same JS context as literally everything else, no processes, internal FF methods were being patched by extensions, etc.). Their power came at a great cost and their approach is inherently infeasible in today's security conscious world (even if you ignore the stability and performance issues).