But the "observer" can still know which mobile phone is yours and who communicates with whom? Especially if the "observer" has the info from the Signal servers.

Edit (as i can't post you reply to your answer):

And based on the NSA principle of the "thee levels of distance" everybody is reachable as long as some common numbers are in our contact lists which we happily upload.