A bloom filter of my contact’s phone numbers that is updated on my device and checked on the server could be a good privacy approach. A small number of false positives (robo calls getting through) would be acceptable if the device still filters or presents the option to accept.

PINs could also be a bloom filter on the server, then the device filters on whether the specific PIN entered was single use or has been revoked.