Anyway, people can get IP addresses of phones, public Wifi still tells them where you were and it only takes on mistake of you connecting through your own phone for them to know certain.

This taken together means there's no security with regard to contacts if the server or the network is compromised.

There are excellent solutions to this: dummy messages and remailing. Without those no system can ensure that contact information is kept secret.