It wasn't working well, and it was as much a walled garden as the chrome approach. XUL extensions were/are very strongly tied to mozilla, and allegedly a cause of its enormous memory leakage.

If you want an open approach look to userjs (e.g. userscripts.org); with a little bit of effort you can make a script that works across FF/chrome/opera/probably others. Of course, you'll still find idiots writing things that only work in firefox.