Well not quite, since you might have had their number but they not yours. But if you just talk to someone on Telegram, they are not automatically added to your address book, and thus can't see it by default (as I understand it)