A privilege escalation in MM daemon would still allow you to write and read any user memory. Just not kernel memory or execute anything not covered by memory access capabilities. For nearly all intents and purposes, it is root.

Couple things here.

Firstly, as the MM daemon runs on its own process and is well-separated from other code, it is far easier to audit, debug and so on. Its interface is also entirely explicit. There's value in modular programming. It's far more reasonable to expect quality from such a MM daemon than the mess in a random monolith kernel.

Secondly, in seL4, physical pages are capabilities. There might be more than one MM daemon, owning separate sets of capabilities to physical pages. Security-critical memory might be managed by a MM daemon your vulnerable process has no capability to talk to.

Just my two cents.