Can you think of other online systems that need to scale to massive numbers of users on day one of it's release? If you can, how many of those systems survived?

With my own game project, I used around 50,000 bots on AWS and attempted to model them as closely as possible to real players (using real player data taken from our Closed Beta). Looking at resource utilisation on the servers, it looked like we should be able to scale even further than those tests. By that I mean, the most overloaded systems were still only using about 35% of the system resources.

Come open beta day, we got concurrency of around 75,000 at peak, higher than the optimistic estimate we had made of 50,000. New problems developed. In fact, new problems developed at around 45,000 that somehow the bots didn't manage to find. The worst problem was a deadlock condition that could take out one of the database servers randomly kicking off ten's of thousands of users at a time. It's hard to describe the feeling of knowing that you are frustrating so many people around the world.

More servers would have done nothing to prevent these problems.

Anyway, I guess what I'm saying is that scaling all at once is really hard. Even if you take lots of precautions, random problems will still develop.