Matrix's anonymity comes from the fact that anyone can create an account on a public server on the network using an arbitrary username that doesn't have to be linked to a phone number or email or any other identifier. However, the server they use can see their IP address and their access patterns, and the other servers in their rooms will be able to see what rooms they are in and the metadata of their message history in those rooms. This could theoretically be used to deanonymise users - after all, the rooms where a user hangs out, the members they share rooms with, and their message frequency in those rooms is all sensitive metadata. As tptacek points out, Signal doesn't track that and just passes messages from A to B, whereas Matrix by design is about syncing conversation history between users & servers, and to do that it needs to record the conversation history (and thus its metadata).

We're mitigating this with P2P Matrix (https://fosdem.org/2020/schedule/event/dip_p2p_matrix/ etc), where there are no longer servers where metadata pools.