News for hackers. Stuff that matters.
Re:WebSockets, Erlang and shattered glass by wulczer on Monday March 26, 2012 @03:31PM (#3758263)
We (hope we) did learn.

First of all, pay closer attention to the usage patterns. We saw increased activity for a few days before the beta signup, but never investigated in depth. We only learned that something's up when the heavy traffic started and monitoring alarms went off.

Second, get the scaling-up procedure more automated, we had everything planned, but it took too much time to get the extra infrastructure ready. Pressure and precise operation don't work well together.

Third, we uncovered a previously unknown memory leak in one of the components, which got evident during the Guild Wars 2 event and forced us to restart a subsystem a few times.

All in all it was a nice (if a bit stressful) experience, hope you guys enjoyed it a bit yourselves too :)

Re:WebSockets, Erlang and shattered glass by endianswap on Monday March 26, 2012 @07:43PM (#3759185)
Thanks for the details. I'm not going to say it was awesome, but it was pretty cool to see our fans accidentally DDOS a service just by sheer excitement. I mostly felt bad after I looked up Ducksboard and saw that you guys are a pretty small operation; that made me feel a little bad. ;)