If they can come in at any time and say "we only want this one guy" then the rest of the users privacy is already in trouble.

In a sense, yes. One of Lavabit's failings is that they didn't design for scope-limited disclosure. When the FBI demanded their certs, it exposed all their users.

Don't overthink it. The failure of Lavabit was designing a system that enabled disclosure at all. It wasn't end-to-end encrypted; its users had to trust its operators, which, because operators can always be coerced, is never safe.

Not familiar with the lavabit story - how did his refusal cost users their privacy?

He had to give up other peoples' privacy, as well. The alternative (that the FBI demanded until he started playing childish games) was actually only handing over Snowden's mails.

Which, to be clear, would still have been a disaster for Lavabit! It's not my suggestion that he should have gone along with the FBI and then continued to run the service.

But had he complied, as he was legally obligated to do, with the FBI, he could have immediately and gracefully shuttered the service without having given up the TLS keys that unlocked all communications to the site.

I can't remember whether the Lavabit story was before or after the leaks. Would we still have the Snowden leaks if Lavabit had complied immediately?

It started after the leak (the first pen/trace order was from June 2013).