Signal does not tell me which number is which user. I know which number is who myself. The most Signal does is presumably warn me when the key associated with the number changed (eg. new phone).

And that's where I have to trust Signal, but as a protocol not a "trusted source" of information.