As I pointed out elsewhere in the thread, Signal uses SGX to let clients confirm exactly what version of the contact discovery server software they're running against:
https://signal.org/blog/private-contact-discovery/
It's not bulletproof by any means (https://signal.org/blog/private-contact-discovery/ goes into this, and also points out several features they've used SGX for since), but it's certainly something, and they're doing it.
Where does Signal prevent you from using your own server? Spin up your own and use it if you'd like.
We are very much trusting the Signal team to do what they're saying (not logging, not leaking) and to make no mistakes in doing it (not logging accidentally, not leaking accidentally).
Which is in general a fair trade-off, but it is very much a trade-off, open audits notwithstanding.