AFAIK they don't encrypt the contact list, they hash each number and send a few bits of the hash to the server. The server replies with a few possible hashes it knows about, so the client can reason that "if the hash I was asking about is in that set, the user I want to talk to has Signal installed".