This is sort of true. The source is published and you can build your own binary. But given that you can't distribute Signal outside of official stores and can't pin the version in those official stores (unless you turn off updates on your phone entirely), it's not actually practical to run an audited version, yet alone to make your own changes to the code.

> and that is was allowed to run your own servers. Are the servers even open source?

EDIT: apparently there is now (purported) server source available, not that that means much when there's no way to even know which code a given server is running, yet alone run a server with different code. They claim that their E2E encryption means control of their servers doesn't matter, but their protocol analyses doesn't actually think about what an attacker might be able to do at the server level, IME.

> Are there lirerature regarding the technical/conceptional bits Element/Matrix? What is the tradeoff there?

It uses either the same ratchet protocol as Signal or a very similar one. E2E for group chats is more complicated but I don't think you're giving up anything.