iCloud contacts, which is how Apple phones store and sync this metadata for you, is not end to end encrypted, and is available to Apple and the military intelligence community without a warrant.

https://sneak.berlin/20200604/if-zoom-is-wrong-so-is-apple/