The internet is full of half-baked "secure" and "private" chat clients and servers. Security is hard to get right. I'm guessing they don't want the reputation of Signal muddled up with some other random client. IIUC anyone can run their own network, and make it as open as they like, using their code, they just can't call it Signal. Much like how Firefox and RedHat use their trademarks.
For all we know, both the Signal network and its client are a half-baked "secure" and "private" chat, and the source code they publish is an elaborate decoy (though probably a subset of the real underlying code, for obvious reasons).
That's a valid question which used to bug me about open source projects. But apparently they finally figured out that the output of the source needs to be deterministic and match the binaries they ship. This property is called "reproducible builds". Signal claims to have them (modulo some third party libraries), though I haven't personally verified it: https://signal.org/blog/reproducible-android/
Honestly though, trust boils down to trusting people. I trust Signal because I trust Moxie, and I trust Moxie because of his reputation among the prominent security experts publicly active on the internet, at least the ones that I find convincing. As a security layman, that's the best I can do.
Your "problem" wouldn't be solved by allowing third-party clients. You still don't know what the server source is.
I don't think I'm the only one though, at least to a degree. When the only thing you allow your users is to read the source code, you are technically open-source, but it's the kind of openness that doesn't guarantee anything. More and more often companies use the term "Open source software" because they know that people will walk around the internet and say "it's safe and secure: after all, it's open source!"
OWS is aware of this, which is especially valuable when you brag about privacy as a communications network, but they deliberately chose to limit what you can do with said "open source" – to a degree that's quite ridiculous for a company that calls themselves "Open (...) Systems" :)
You're mixing up rights to the software with rights to Signal's name and infrastructure.
Open Whisper Systems is a company that uses "open-source" as a marketing term while bullying the people who choose to take advantage of it.
You can build your own client, but if you start to distribute it, they'll ban you from using their servers[1], with an explanation plain ridiculous for a company that took thousands of dollars from various open source foundations. The server is open source, but again, you have no guarantee that the official (and only) signal server is anything close to the software they published as open source.
Telegram at least publishes libraries that you can actually use with their official service [2]. You still have to „trust the company” to a certain degree: you'll never get proper security without something like Matrix or XMPP where you can be in control over both ends of the communication. Still, a company like OWS or Telegram can do more or less to put users more in control over their end: and I'd argue that OWS is doing way less for that than Telegram does.
[1] https://github.com/LibreSignal/LibreSignal/issues/37#issueco... [2] https://github.com/tdlib/td
EDIT: removed "dishonest" – I wish I had strikethrough to use :)