Source code is merely convenient for the security community in cases like this. But it's hardly a requirement at all, and you can absolutely build a secure system if you design for it.
All that said, I think source code for a device as personal as a phone is largely more of a question of ethics and this is what Purism heavily leans on as an angle for their marketing. I think that's a good angle, independent of any security aspects: owning the code and having the right to it (even if you don't ever touch it) is an important aspect of owning a device as deeply personal as a phone. People like us who believe that should totally advocate for it -- I think it's a fairly simple case to make. But it really has little to do with _security_ on its nose, if you ask me.
(Also, if your government mandates backdoors, they will not be thwarted so easily by simply recompiling AOSP or whatever and loading a new ROM, if they do not wish to be. Any belief to the contrary is a fairy tale that people like us enjoy telling themselves. But I'll give you a hint: writing a Makefile cannot get you out of jail, and you are not that good at covering your tracks.)
Realisticaly, though, a manufacturer or intelligence agency will be the one paying security people for thorough reviews. Anyone doing this in the open would do this to build a reputation or for their own security, and will need much more time to review something that first needs to be reverse engineered. They will almost certainly not be able to look at everything.
It's not a logical argument to say that we security people are used to reading obfuscated binaries anyway and conclude that therefore no fewer bugs will be found compared to the situation in which it is open source.
As governments are pushing more and more to have back doors(or side gates here in Aus) it is becoming very important to have fully open code, as it is the only way to get around the vulnerabilities governments will be requiring phones to have in them.