Server and the contact directory service are two different things. The remote attestation is just for a small part of the code (SGX code have a lot of constraints, including size). Also, I don't know if it changed but the new contact discovery service was not in production last time I checked.
Note: my understanding of SGX is that this still requires trust of Intel. Still, far better than nothing
It also requires there to not be a hardware sidechannel exploit on the device running SGX... such as the variants of Spectre that have ben trickling out every few weeks now all year... so yeah: Moxie's trust of SGX is pretty damning.
Signal's server code is open source as well: https://github.com/signalapp/Signal-Server
And apparently the client can verify that the server is running that code: https://signal.org/blog/private-contact-discovery/#trust-but...