That's all moot though, since your server has access to plaintext emails when sending and recieving (99.99999% of email addresses would be outside skiff), which completely subverts the whole point of encryption. A vulnerability on the server could leak all user emails, without needing their keys.... This is a solution theoretically only as strong as encryption at rest.