Wait—when you're using Signal, it resorts to using your whole-phone contact list when, say, you want send a direct message? That would be... not great UX, with my personal use of my phone contact list (mostly for people I barely ever message, contacts I'd never message but want to have a phone number and/or address down, or relatives who don't/can't use anything but SMS)