Using Matrix+CRDTs for this is a pretty cool idea! It'd take a rewrite of this app, but you could let arbitrary Matrix chat server host the entire data store, getting rid of a lot of the data security risks (as long as you make sure your e2ee approach isn't too eager to share keys).

Then again, doing things this way might make data deletion and other privacy related issues quite difficult to achieve, especially if said Matrix servers are federated.