Now, a more interesting privacy concern is that Matrix doesn't and can't protect metadata currently - see https://matrix.org/~matthew/2015-06-26%20Matrix%20Jardin%20E... for details. So if you want both encrypted contents as well as obfuscated metadata, go check out Ricochet or Vuvuzela or Pond or something for now :)
For now, building a great federated chat protocol that allows users to fully control their own data should take priority over just about everything else, because once users are in control of their own data, they're no longer locked to a single platform. So if another protocol comes along that does everything Matrix gets right and adds metadata protection, users can simply export their data and import it to the next great app/protocol (which is hopefully also federated). Until we have a great federated protocol that everybody is using, the friction associated with moving between chat apps/protocols is going to continuously inhibit real innovation in the messaging apps space by favoring apps that stand on the strength of their network effects over apps that stand on their own merits.
On the topic of fully distributed apps, I really like what replikativ is doing in this space:
https://github.com/replikativ/replikativ
It's a CRDT-based data replication library for building fully distributed applications. They envision replikativ being used as an open data exchange whereby users fully control their own data, and can specify any number of applications to have access to that data, instead of the status quo where user data is kept in closed, per-application silos. Their README has a much more comprehensive overview of the library and their vision.
I'm hoping it's still on some sort of roadmap.
I definitely appreciate that the Matrix team is aware of and actively working on E2E encryption, but it doesn't change my stance that Matrix is not a feasible building block for privacy-minded apps until that work is complete.