AFAIK they don't encrypt the contact list, they hash each number and send a few bits of the hash to the server. The server replies with a few possible hashes it knows about, so the client can reason that "if the hash I was asking about is in that set, the user I want to talk to has Signal installed".
What they're moving towards is a design that looks like what Apple did with their HSM quorum system. The contact information we're talking about is encrypted clientside, but with (usually) a memorable pin. Without countermeasures, memorable PINs are very easy to attack; SGX allows them to artificially limit guesses. As a user, you retain a security dial on this: you can use a more complicated passcode than a 4-digit pin if you don't trust SGX.
Obtaining the whole database Signal maintains gives you ciphertext that you need to mount attacks on user-by-user (and to make those attacks, you'd have to break SGX). It doesn't simply give you the plaintext SQL database other messaging systems collect.