1) The server operator could choose to obtain access to plaintext. 2) An attacker who compromised the server could get access to plaintext. 3) Anyone capable of intercepting the SSL communication could get access to plaintext.
Incidentally, those are the exact same points of vulnerability for a normal (unencrypted) mail service.
One interesting question is why the US government requested Lavabit's SSL key rather than just getting a CA to sign their own. My assumption is because they were interested in past communication that might have already been deleted (perhaps by Snowden). We now know that the US government often logs and stores ciphertext, and we know that Lavabit was not selecting PFS SSL cipher suites.
So when Ladar did eventually provide the SSL key to the government, it's likely that the government was able to use that to decrypt all previously stored traffic and obtain the entire history of transmitted email.
So it's quite likely that Snowden (and all other Lavabit users) did have their email read by the government.
Email just isn't very secure as a protocol; people that care need something they can verify, like PGP (which Snowden used). It's unfortunate that the asynchronous nature of email (which we tend to view as an important feature) is incompatible with client-verifiable forward secrecy.
BTW: shouldn't we assume, that the US government operates at least an own intermediate CA for such purposes rather than getting single Certs signed.
The technical challenge is in designing a system in which the server does not have to be trusted, but that integrates well with existing protocols. I'm not aware of a proposal that offers the desired level of security while transparently integrating with SMTP, but perhaps Moxie has ideas.
One can get part way if the users generate their own key pair and only provide the server the public key, then either use a PGP-capable mail client or perhaps run a local (open source and audited) IMAP proxy to decrypt. You'd still have to trust the server not to store a clear copy of incoming mail, but once it is encrypted, only the client would be able to decrypt.
If you need the server to be able to text search through your mail archive (and I do that a _lot_ on my phone), you need to "trust" the server with your clear text.
If you don't want to run and secure your own server - you need to choose what level of compromise you're prepared to accept.
My current project is a mail server on a RaspberryPi, hidden behind my home NAT gateway and firewall, that opens reverse ssh tunnels for ports 25 and 465 to inexpensive VPSes configured to not log anything - with IMAP (and webmail) access limited to my local network (and to my phone and iPad via vpn). It uses Vagrant and Ansible to provision and configure VPSes on the fly, and DNS hosting with APIs that Ansible can update MX records for as my receiving mail server moves around different VPS vendors in different jurisdictions.
How many megabytes of body text are you searching through? Is your phone actually incapable of doing the search in milliseconds?