Per Mozilla, aside from security problems and other nightmares, these deep hooks allowed developers to write plugins that were incompatible with a multiprocess browser architecture where each tab/window can have its own sandbox and remain responsive even if other tabs go down.
And that is one of the killer features that has caused FF to bleed massive amounts of user share to Chrome in the last decade: Chrome almost always "felt fast" in situations where FF ground to an embarassing halt because something was happening in another tab/window that was causing the entire browser to become unresponsive.
I don't know what precisely is missing from the new WebExtensions API but I have to imagine that Mozilla will work to expand its capabilities as much as possible over time...
The Add-on SDK was supposed to fix the "deep hooks" problem and is mostly multiprocess-compatible. As far as I know, most currently maintained XUL extensions are multiprocess-compatible, too.
I've seen plenty of my favorite software projects have to make "hard breaks" to modernize. KDE broke everything with Qt5 / Plasma 5, but the payoff is now they are the premier high DPI Linux desktop. Python broke everything with 3, but features enabled by the changes have made Python 3 the best scripting language ever in my book.
Firefox would be doomed to a slow death with its legacy extensions API the same way PHP and Perl languished for years with the inability to push a major version. A slow death is not a downfall anyone wants.