I have not looked into how it works, other then reading the blog post, but it seems like they are trying to figure out that part as well: https://signal.org/blog/sealed-sender/
But yeah, as long as they own the servers I am sure they could puzzle it together if they wanted.
worth looking that the cwtch.im alpha chat app, that attempts to mitigate the metadata problem with TOR, and having untrusted servers that host group chats.
Similar philosophy to Signal of trying to really get usability right (looking at you, Element). Though still early in alpha development, wouldn't trust current alpha builds to be reliable.
This data is there at any rate however confederated chat apps can use different servers (so data is not congregated) and you can also change accounts easily (it's harder to change phone number).