These people tried something like your idea.
If you're going to do something like what they did then in my opinion it isn't really feasible. For starters there are scalability issues. Also, if you're concerned about your privacy but want to message someone who doesn't have this app then you're forced to message them insecurely. I remember having some other reservations about this when I read it but it's been a few months and I've forgotten them.
These people tried something like your idea. If you're going to do something like what they did then in my opinion it isn't really feasible. For starters there are scalability issues. Also, if you're concerned about your privacy but want to message someone who doesn't have this app then you're forced to message them insecurely. I remember having some other reservations about this when I read it but it's been a few months and I've forgotten them.