Matrix's anonymity comes from the fact that anyone can create an account on a public server on the network using an arbitrary username that doesn't have to be linked to a phone number or email or any other identifier. However, the server they use can see their IP address and their access patterns, and the other servers in their rooms will be able to see what rooms they are in and the metadata of their message history in those rooms. This could theoretically be used to deanonymise users - after all, the rooms where a user hangs out, the members they share rooms with, and their message frequency in those rooms is all sensitive metadata. As tptacek points out, Signal doesn't track that and just passes messages from A to B, whereas Matrix by design is about syncing conversation history between users & servers, and to do that it needs to record the conversation history (and thus its metadata).

We're mitigating this with P2P Matrix (https://fosdem.org/2020/schedule/event/dip_p2p_matrix/ etc), where there are no longer servers where metadata pools.

Signal's sealed sender reveals ip addresses of the sender as well. And if you use the same IP to log in to the signal service, they can figure out who you are easily. The data pools even more as Signal is centralized. You only have to get AWS give access to Signal's infrastructure. And guess what, the DoD has giant AWS contracts already and with that tons of leverage.

Wouldn't using something like tor mitigate the issue of servers seeing IP addresses?

sure, or any kind of anonymising vpn.