The important part is that you are able to look up/personally verify that you are contacting the correct people. If I lose my card I can still look up the right number on my banks website. Because you cannot confidently identify a caller it is better for the customer to reach out using more easily verified information.