This also destroys Don't Stand Out. Conspirators using Signal to plan don't stand out from people talking about their Buffy re-watch party, discussing what sort of fast food to order, complaining about their jobs... it's all just encrypted flows to and from Signal's servers. Whereas once you deploy your super secret server for a federated service you actually just flag all the users of that server for attention.
With the Signal protocol this isn't a thing. Where metadata has to live on Signal's servers (e.g. "new-style" groups where you can have moderators) it's encrypted and the servers can do operations on the metadata without understanding what it means in a human sense. They go to considerable lengths to ensure they have no idea who their users are, who they talk to, or what about.
With Matrix you have to choose, Stand Out by setting up the private server which may attract problems, or Don't Stand Out but give away your metadata to the big public servers and assume the Secret Police have it.
Element provides better privacy in theory but to get most out of it you'll need to set up your own server, which isn't for everyone. Otherwise it's really more of a question whether you trust the Signal foundation or any of the public Matrix instances.