IIRC it only works if you are on their contact list. And I think you need to be in the settings app. Something like that. It's a handy feature but Apple could make it easier to understand, and they could do way better communicating why it isn't working, when it does not work.