Warning:
    This section documents the state resolution algorithm as implemented by Synapse as of December 2017
    (and therefore the de-facto Matrix protocol). However, this algorithm is known to have some problems.

[0] https://matrix.org/docs/spec/

[1] https://matrix.org/docs/spec/server_server/unstable.html#roo...

* implementing E2E encryption and getting it audited, which turns out to be very hard in a decentralised environment

* refining Riot as a flagship client and trying to get it to a Slack/Discord level of UX (still a work in progress)

* scaling, as the 1st gen implementation (synapse) failed to keep up with the growth of the network

* adding features like Jitsi video conferencing, Widgets, Communities, Home pages, Rich Text Editing, Read Receipts, Synchronised Read Markers and much much more.

* dealing with a major funding crisis last year.

In retrospect we probably should have spent more time finalising the current features rather than adding on new ones, and since the end of last year we've been feature-frozen in order to finish the stuff in hand and get to a 1.0.

Fixing the issues which remain in federated state resolution is probably the hardest problem to solve before a 1.0, and I'm happy to say that over the last few months we've made major progress in doing so.

Agreed that in an ideal world this would have all happened sooner, but turns out that decentralised open protocols are harder than centralised startups like Slack and friends. Who knew?

Do you have any public discussions or blog posts about this decision?

The difference for a project like Matrix however is that our product is not really any single software codebase but the protocol spec itself. In fact, it's a bad smell about the spec if it's too hard to implement it - whether that's clients or servers. So part of the point of the Dendrite (golang server) project is to dogfood the spec and fix its various shortcomings (as per the OP's complaints) and make sure it's fit for purpose. For context, the core Matrix team has already written 4 entirely disjoint Matrix client codebases (Matrix Console on Web, Riot/Web, Riot/iOS and Riot/Android) as part of developing the spec and ensuring it's fit for purpose, so it's not that unreasonable for us to also write 2 server implementations (3, if you include the failed Dendron project).

In terms of blog posts: https://matrix.org/blog/2017/03/15/dendrite-receives-its-fir... is probably the most revealing one.

It's worth noting however that Dendrite progress has not been entirely smooth, though. Particular problems have been:

* When we lost funding last year (https://matrix.org/blog/2017/07/07/a-call-to-arms-supporting...) it inevitably had an impact on the team - and ironically the only two people who left the project ended up being the two folks who were working on (and created) Dendrite. This genuinely wasn't a reflection on Dendrite itself (which is ironically one of the most fun projects to work on in Matrix, given it's all blue skies and green fields in terms of implementing a server that learns from Synapse's many mistakes), but more a sad coincidence. Either way, it ended up with the project being under-resourced (although we're finally fixing that during May having re-hired :)

* Matrix traffic accelerated massively during 2017, and Dendrite wasn't yet ready to use in production - but meanwhile Synapse wasn't performant enough to keep the matrix.org server running smoothly, so we had to pull folks off Dendrite in order to keep Synapse scaling (ironically, ending up taking inspiration to some extent from Dendrite in improving the Synapse codebase)

* In general, it's hard to make progress on an R&D project which isn't yet in production when there's an existing project in production which is on fire.

* We deliberately scoped Dendrite to implement a server against the current Matrix spec that Synapse implements - rather than trying to improve the Matrix spec at the same time. This is good in terms of feature creep, but possibly a bit demoralising as you find yourself failing to innovate and improve the spec and instead re-implementing it, warts and all. I'm not sure what the right balance is here, and in practice we've ended up with the Matrix spec evolving significantly under Synapse, and meanwhile Dendrite then becomes a moving target as a result.

* We've almost certainly fallen foul of the Osborne Effect (https://en.wikipedia.org/wiki/Osborne_effect) too, where folks have held off on using Matrix or running Synapse because they're waiting for Dendrite to land, RSN.

But on the plus side, there's a small but active FOSS community contributing to Dendrite - much more than we've ever had on the relatively impenetrable Synapse codebase. And Dendrite itself works (about 80% of an MVP complete) and is demonstrably ~2 orders of magnitude faster than Synapse (even whilst Synapse's performance is also speeding up). And as we find algorithmic breakthroughs in performance as part of improving Synapse, these breakthroughs equally apply to Dendrite and will make it even crazier fast.

So, I think it's too early to say whether this is a textbook example of "second system syndrome", or whether instead it's a mature and sensible approach to ensuring a heterogenous ecosystem of implementations for an open standard and dogfooding one's own protocol. My hunch is that if we hadn't hit funding nightmares last year it would be a no-brainer win; but even now, it's likely to eventually work out for the best in the end. The best analogy is not the excruciating migration from Netscape to Mozilla that Joel wrote about in 2000, but the more spectacular and successful migration from Gecko to Servo happening in Mozilla right now. It's a massive massive architectural improvement and rewrite in a new language which requires a lot of work, but pays off in the end.

(This is my personal take on it, at least, as project lead for Matrix; those actually working on Synapse & Dendrite probably have different viewpoints! :)

https://github.com/matrix-org/dendrite/graphs/contributors paints a fairly clear picture of where things have got to (with Synapse temporarily taking priority since Jan).

However, in an open source project rewrites are often successful. Sometimes it frustrates the users in the process (see GTK/Gnome/KDE rewrites) but often there is not enough pressure to kill the rewrite so it succeeds. Whether or not you'd be better off with the original is often an open question, though.

I don't know enough about Matrix/Riot to comment about whether their rewrite was a good idea. I hope they are happy with it.

Highly successful project. Sometimes a rewrite is the right choice.

//this seems to work now, not sure why, but it does, should really look at this in the morning 2012-05-12

and General experience [2]

mostly because legacy systems contain a lot of implicit features and undocumented decisions that are hard to re-discover and customers get really annoyed. Maybe less of a problem in this case since it is young and rather well defined domain.

[1] https://en.wikipedia.org/wiki/Second-system_effect

[2] https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Any comment on that?

> Preparation for py3 (PR #3061, #3073, #3074, #3075, #3103, #3104, #3106, #3107, #3109, #3110) Thanks to @NotAFile!

https://github.com/matrix-org/synapse/releases/tag/v0.28.0

Just wondering about discovery. How do I find interesting channels?

The list (and graph!) of all indexed rooms is here: https://t2bot.io/voyager/

Given that matrix describes itself as an open standard, neither of those aspects says much about the project other that there's either some goodwill or a PR department.

All of the Matrix/Riot users I know use private, non-federated home servers, but still rely on the project's Sygnal server(s) for push notifications.

[0]: https://en.wikipedia.org/wiki/Minitel

[1]: https://www.forbes.com/forbes/1997/1201/6012058a.html#75b6a6...

Specifically from [1] is the line: "Its illegal in France to encrypt any communication in any way unless you have permission from the government. How is electronic commerce going to get off the ground in such a society? Good question, which appears not to have occurred to the authorities."

[0]: https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFT...

[1]: https://en.wikipedia.org/wiki/Cryptography_law#France

I'm looking at you, Xilinx.

    Criminal Procedure Code: Article 230, modified by Law No. 2001-1062 
    and Law No. 2014-1353 requires the disclosure of encryption keys upon 
    the authorization of a judge.

Competitor here, few questions:

- You guys are currently implementing federated protocols, right? We do P2P protocols, I'm curious why you guys chose federated - why do you think it is better?

- How are you guys different than Signal? Obviously way different than Telegram, as it isn't even Open Source (thanks for Open Sourcing your work!).

- When I was working on our E2EE private messaging app (ugly demo here: https://twitter.com/marknadal/status/989602258638684160 ), I ran into problems about metadata. Stuff like, do you guys reveal which public keys are talking to which other public keys? And the frequency of those conversations? Or do you encrypt that information as well (and if you do, how do you handle discovery then, and is it vulnerable to network frequency analysis attacks?)

Awesome work, congrats again, keep it up! Curious to hear more about your architecture.

afaict, Matrix is both, kind of. Rooms are decentralized, while users are federated.

> What about signal?

Signal is great, but it solves a very different problem. It's a good replacement for talking to your best friends on your phone.

But it fails quickly once you go past that. No true multi-device, no talking without releasing your phone number, no public groups. No mentions, no permalinks. In that regard, I'd say matrix is closer to telegram and irc than WhatsApp or Signal

> what about metadata?

The state of metadata in E2E is... not good. Really, only the content of the message is secret. There are a number of ideas how to deal with this.

Riot does this pretty well, and when it's not alpha/beta, it'll probably do it really well.

For instance: at the moment Matrix server implementations expect clients like Riot to long-poll for JSON about each room/chat they're connected to. Alternatives like Prosody's XMPP server implementation dropped websocket support in favor of BOSH years ago.

I wonder why this is - maybe there's just less developer interest since chat has gone mobile, and mobile devices break ws connections so frequently? But it's obvious that chat clients like FB messenger, whatsapp, slack, etc are establishing ws connections.

Strange.

You probably refer to the "old websocket" spec, new websockets are working well: https://prosody.im/doc/modules/mod_websocket

Quite the opposite, but I admit the documentation is in need of improving to make the websocket functionality easier to discover.

It is in the default config file however, right next to BOSH.

Sending messages though from a server to a room was more difficult than I expected. Java SDK didn't build and many of the other languages were alpha too. Go seemed favored but I dont really want to install. Is there a recommended simple command line app or script to send messages?

What do you mean? If you mean Go, the programming language, you don't need to install any of it to use a Go application since those are statically compiled.

The preview docker image of mattermost is easier to install but I installed matrix because I couldn't get the production docker version of mattermost to work.

I wasted most of time configuring the vhost and the SSL certificates.

which java sdk did you try?

Huh. So Haskell + Servant might be really nice for writing a client, then, yeah?

What does WeeChat offer over irssi? Very active development, scripting in many languages, dynamic filtering, raw buffer views, sane defaults, helpful configuration interface, good documentation, live/hotswapping upgrades, architectural soundness, many options for interaction (FIFO pipes, client-server protocol) and so on and so forth. I'm always surprised at how little WeeChat gets in my way when I want to do something out of the ordinary.

And unless you are heavily invested in the irssi ecosystem, the switch is basically effortless.

That's the cost–benefit of it. Another way to view it is this: People who switch from Irssi to WeeChat keep using WeeChat; while people who switch from WeeChat to Irssi go back to WeeChat.

I used the Java SDK from Kamax that was on https://matrix.org/docs/projects/try-matrix-now.html. would be nice if there are others.

I'm excited about it though - is just what I've been looking for.

(Note that in first world countries phone calls will be digital everywhere except maybe the last mile, and GSM and upwards are digital to the handset)

Presumably, if there's an investigation, they just force the government employee to give up the keys.

http://www.conseil-constitutionnel.fr/conseil-constitutionne...

http://www.ucl.ac.uk/constitution-unit/research/foi/countrie...

Do you mean IP lawsuit's against Matrix itself?

There's not much that Matrix can do to defend against that, except for strategic marketing with the purpose of achieving a large distributed userbase. If Gmail had existed in 1990, we would not have a decentralized e-mail network today. If Gmail had got into a position where they served 99% of all e-mail users, they would've pulled the plug on mail delivery to third-party mail servers and e-mail as we know it would've been dead.

Not that I'm glad that most people I email with use gmail, since I'm precisely hosting my mails to escape Google (well, this is one of the reasons).

I cannot say that for Microsoft. After a long discussion with them, I think I'm not systematically landing in my recipients' spam folder anymore, but I'm still not very confident. Microsoft, please fix your filters! If I already exchanged with somebody in a two-way direction several times, my mails should not land in their spam folder anymore, especially if they explicitly marked me as not spam (even more if they added me to their contact list), and if you observed only legitimate mails for months or even years from my IP/domain, my mails should not be treated as spam by default even for new recipients, right? Good and reasonable filters are important for internet neutrality, where small providers can exist.

I don't know much about Matrix or Riot.im (from what I remembered, it was a implemented IRC alternative), but if it delivers on all of the above (and the article sure sounds like it), then it's definitely polishing up some final features that were still missing!

You can even send E2E-encrypted messages/files to people who haven't even joined yet!

https://keybase.io/blog/keybase-chat

The server is not, and the Android code is updated months after each update, all commits being squashed(see issue[1]). Also their new app (Telegram X) is not open source.

[1] https://github.com/DrKLO/Telegram/pull/1431

Not true, it has a websockets alternative to gcm.

>and requires a phone number

The phone number can be anything you control.

>and turned-on phone iirc

Yes

>(and their profit model seems to be extorting companies like Wire and helping closed source software like Whatsapp advertise with Signal-level encryption).

It (now) has its own nonprofit backing it (and a generous benefactor). It's not trying to be profitable.

The "extortion" thing seems to have been about wire using signal code in breach of copyright (ie not complying with the open source license).

Signal has stated that the goal of working with WhatsApp (and others) is to make good crypto be used as widely as possible. Matrix crypto is also based on what's used in signal.

>Wire is the best of all, but the Android app drains battery and everything is web

Signal is very polished (at least on Android - I'm not familiar with the ios app).

TLDR: If you don't control the server, there is no security guarantee. So signal, whatsapp and telegram are out.

WhatsApp is the same but has things like key-change messages off by default.

I believe telegram doesn't even encrypt by default, and has flawed crypto anyway.

You're right that the whole thing is flawed to some degree unless both parties are clued up.

Edit: If DNSSEC is as secure as sms, then signal is as secure as sms.

Not message contents etc.

related commit: https://github.com/signalapp/Signal-Android/commit/166973132...

pretty difficult to find the raw apk though: https://signal.org/android/apk/

about matrix:

you will probably end up having to use the electron riot app for a while, but the community is working hard on native clients, but afaik no client has e2ee in any usable state implemented yet (but a lot of progress is being made).

I think the matrix.org team acts as a non-profit organisation, but most of the core team is also hired by New Vector where, I think the planned profit model is to run matrix instances as a service.

there's some stuff about the funding in the faq: https://matrix.org/docs/guides/faq.html#what-kind-of-company...

I tried it less than a year ago, because I had heard that too, but it still didn't work. I should try again on my new phone I guess.

https://signal.org/blog/whatsapp-complete/

And that's low hanging fruit.

I can't wait to see the end product (this summer 2018) :)

especially considering they "forked" it, who knows what they will do with it (log keystrokes, take screenshots, in the disguise of diagnostic reasons?).

anytime you are reading a message from your app, or sending a message, the app have all the powers to capture the data in its raw form before transmission, encrypt it with some key that only they have control of, etc. so unless the gov't releases their source code to be truly open (which even then i don't see how you will ever know), you won't get the transparency you need.

encryption is such a buzzword, people think anything encrypted means you are fully secure. this is simply false.

    We understand the whole project is going to be released entirely open source (other than the operational bits) 
    – development is well under way and an early proof of concept is already circulating within various government 
    entities.

At some point, if you work for the government you have to use and trust the tools at your disposal. You are part of the executive power, after all.

Is the application they're creating also going to be open source and available for auditing?

thereby, Riot is developed by the france government. Big change it has a back-door or something. you cant trust it as a citizen. correct me if i wrong but this sound like a honey pot to me.

Source code here: https://github.com/signalapp/Signal-Server

If anything, that'd be a feature, no?

I ran Synapse on a resource-limited machine, so it had sort of "lagged behind" (or something like that) a little bit now and then. And I saw a number of "error decrypting image" and similar issues. It had self-resolved somehow after a while (saw the message decrypted the next day) but that's still a problem.

Haven't reported this because I have no idea how to collect any useful information.

Do you have link or source for that? (I'm always interested in listening or watching stuff from Moxie)

https://signal.org/blog/the-ecosystem-is-moving/

https://whispersystems.discoursehosting.net/t/how-to-get-sig...

https://github.com/LibreSignal/LibreSignal/issues/37#issueco...

I suppose "only I can do security right" is perhaps overly harsh, but my take away was that he considered a walled garden that could be iterated on by his team preferable to allowing even forks of his team's own client to communicate with users on the OWS network.

>Nothing about any of the protocols we’ve developed requires centralization; it’s entirely possible to build a federated Signal Protocol-based messenger, but I no longer believe that it is possible to build a competitive federated messenger at all.

Fortunately there are solutions that incentivize modern features in federated protocols too, like SSLLabs HTTPS checker or https://conversations.im/compliance/ for XMPP.

- requiere a phone

- build on s3 & a lot of other services without the option of a self hosted alternative.

Look up the signal server requirements.

But even if you’re not in the target demographic, you might benefit: The government will run a large installation and hopefully some code will find its way back to the OS version.

I want to type this into a search bar to see if it's a thing... but simultaneuosly I'm terrified of the prospect of discovering that it is.

It is, or at least Centrelink, Medicare, NDIS and Veteran's Affairs use it in Australia. Wouldn't be surprised if it was also used outside of DHHS.

Matrix has so much potential – someday, when it’s built on technologically more ideal foundations.

Python2.7 you’ll barely notice, but the network overhead of the Matrix protocol becomes heavily noticeable by the end user (especially if your cell phone’s data plan runs out, and you’re stuck on 64kbps until the end of the month, then Matrix in larger rooms just becomes impossible)

And I’m not sure PUT and GET are the simplest solution, I’d think a simple socket over which messages are transmitted in both directions would be simpler than implementing an entire HTTP stack.

but it looks like this hasn't been worked on since october last year.

Matrix is an extremely immature project and the French gov is making an irresponsible move here to prop it up as a solution in any capacity.

It's beta software! Where are the audits? Where are the formal verifications? Not that irc has those, I don't mean to juxtapose one not-good-enough product with another, but this is reaching a point of insanity.

If you want security, use signal. If you don't trust signal, learn to trust scores of world renowned cryptographers, and use it anyways. And if you're still bored, renovate irc.

Gosh this is just crazy. It's like saying the tesla 3 will be the authorized taxi vehicle on Mars. There's so much work yet to be done, it's a million years too early to start regulating and organizing something that's not even half built.

What surprised me is that Matrix is practically a modern IRC spec with e2e in mind. What's better is that the Matrix devs acknowledge this heritage. They don't pretend IRC doesn't exist. They're very up-front about loving IRC, and they have successfully brought over nearly every good feature of IRC into Matrix.

At first I thought it was too good to be true, but it keeps on being good. I have waited for years for a decent IRC alternative which I can convince even my less technical friends and family to use, and Matrix is it. I truly feel like I'm on IRC with my family. It's insanely good. I even use the same client for both.

Matrix can replace your Signal. It can replace your Telegram. It can replace your WhatsApp. It can replace your Facebook messenger. It can even replace IRC. It is that complete.

There is much more overhead to opening a connection and maintaining a connect and sending and receiving, to the degree that people with bad connections are excluded entirely from being able to reliably use it. This is not true with irc. People in third world countries can use irc. If you can ping, you can chat.

We don't WANT a chat for less technical people. We want e2e safety. We should never prioritize """normal users""" over security in a security oriented proposal! That's insane!

But hey, I'm sure youre just a teen with a laptop, so I won't hold it against you that you'd make such an irresponsible decision.

------8<---------------

While the overhead of setting up a fresh TCP connection for every exchange is non-trivial, I'm not sure "reliable on a bad connection" is a characteristic I would attribute to IRC. The protocol is notoriously unreliable to the point where established practise is to run a client on a server somewhere and then connect to that from your bad connection computers. People who don't lose IRC messages sent while their connection crapped out, while Matrix ensures their client is updated to reflect the current state of the servers as soon as they go online again.

I believe also that modern HTTP options let you reuse the same connection for multiple requests, obviating much of the overhead. Not to mention that you are free to extend the Matrix server with alternative connection methods.

We do want a chat for less technical people. In order for humanity to not screw over itself royally, we need to target the 99%, not the 1%. Things can be good and still target non-technical people, as long as they are open and federated.

Of course, convenience does not trump security, but I don't see how that is the case here either.

formal proofs e.g.: https://matrix.org/_matrix/media/v1/download/jki.re/hSmkLkFG...

Agreed that Matrix is still beta, but hopefully we are heading in the right direction and the end is in sight.

This is better that it was before. More security is better. Perfect security does not and cannot realistically exist for a complex system.

I do think have a reproducible binary would be a huge win. Even if you can't verify the integrity of a single build you could verify that build by consensus.

https://nixos.org/nix/

But France does have some domestic defence capability here - Thales make hardware security modules and similar products, and STMicroelectronics has chip design expertise while being partly state-owned.

Apparently, though, people whose threat model isn't dialed up to 11 don't deserve security or privacy?