That's a totally different aspect that reproducible builds are supposed to solve.

Your "problem" wouldn't be solved by allowing third-party clients. You still don't know what the server source is.