Do you?

If you are going to audit something for its security features, you pretty much have to start with a disassembled binary, don't you?

Purely on the fundamentals, you could also start with the published source code, build it, and see whether the output matched the binary you're auditing.

In practice, that's unlikely to work, but it could work.

The problem isn't verifying that the source was used for some particular binary, the problem is that when you read source code, the names of functions and variables will impact your understanding of what they do. If you were to take the source code and remove all comments and randomize every symbol name, then you might be okay.

That's a problem, sure, but it doesn't suggest at all that "real security analysis" means starting with an obfuscated blob and reversing it. It suggests that you're better off doing both kinds of analysis. Variable names can lead your thoughts in certain directions and make it more difficult to see certain execution paths. Obfuscated blobs won't lead your thoughts much of anywhere they weren't already inclined to go -- but they make it much more difficult to see all execution paths.

Signal for Android does offer reproducible builds of the binary now. https://signal.org/blog/reproducible-android/

Not if it's open source and has reproducible builds...

Good luck not being influenced by comments (although they can be stripped) or by how things are named. I still think you are better off starting with a binary.

Which could still be done by compiling the source then disassembling it. You lose nothing by it being open source to begin with.