Even if you only care about "remote phone hijacking", not being able to provide timely vendor blob/drivers/kernel means you're wide open for EoP exploits.
As for why they take such a hard line on physical security, well it's their project and they can have whatever high standards they want, especially if they want to project an image of being an absolute secure phone. The code's all open source so it's not too hard to port to another device, especially nowadays with AI.
Fairphone replaced their own open source OS with /e/ as part of a close partnership with Murena. That's why those are relevant. Fairphones without /e/ still lack reasonable privacy and security, but they're a lot worse with it.
/e/ and Murena have very poor privacy and security for their products combined with very inaccurate marketing. They've repeatedly portrayed private and secure devices as mainly benefiting and being used by criminals and pedophiles. Those are their own explicit choices of words. They've claimed this dozens of times about GrapheneOS including alongside France's national agencies making these inaccurate claims. More information with sources:
If we add that FSF aims to provide free alternatives to some vendor blobs the landscape begins to look promising.[0]
One party to provide needed specs for hardware, one party to free the hardware from vendor blobs and another party to merge this into a working product.