And why should I trust F-Droid's build of Matrix over Signal's build of Signal?
Please understand, I agree with your point. Mine was orthogonal: If you are under threat from motivated and/or state-level actors, using someone else's servers (or clients, for that matter) is a bad idea.
And that includes Matrix. I run my own Matrix server and the users of that server can interact (especially via voice/video) without any fear of being intercepted -- even by me.
What's more, I can't decrypt the conversations folks have in Matrix "rooms" that don't include me without a long process of brute-forcing.
No one is coming to my house to confiscate my server. That scenario is much more likely with a public/commercial service hosted at a data center/cloud provider.
So yes, Matrix is likely more secure than Signal if, and only if you build and install your servers and clients from source with a compiler/linker you built yourself using a trusted tool chain on hardware whose components you've personally confirmed to be free of compromise[0].
[0] https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-tho...
With matrix at least you can pick a server operator you trust to provide your metadata to, or host a server yourself.
I'm not saying "don't use Signal", in fact I still recommend it to non technical people, since it's just much simpler. But pointing at the flaws is a necessary requirement for them to be fixed