If some unstable person is breathing down my neck messing with my contacts is not what I would worry about.

It's too easy to do and can be done from quite far away. Also you don't need eyes. An innocent pic or video taken in public can capture that info.

I think long ago, Signal used to display phone number just beneath name when you are chatting someone. That always made me a bit uncomfortable. Now they don't do that, only displays name.

If they are that close.. they can see the words you are writing. If you are sending winky love icons who you are sending it to may not be as important your actual content.