News for hackers. Stuff that matters.
Re:Skype brief post-mortem by barrkel on Wednesday December 22, 2010 @05:00PM (#2032623)
I can imagine it would self-DDoS from lack of peer supernodes to share the load.
Re:Skype brief post-mortem by dennisgorelik on Wednesday December 22, 2010 @05:04PM (#2032636)
But at least it would successfully reply to at least some of requests, right?

And would still keep trying to serve the requests, right?

So if sufficient number of supernodes is brought back online then the problems should disappear.

Re:Skype brief post-mortem by terryjsmith on Wednesday December 22, 2010 @05:14PM (#2032670)
When this happens to our websites (when all servers go down), we need to rate limit and/or shut down traffic at the load balancer level as we bring things back online, otherwise everything just continues to get swamped and goes right back down. This would be nearly impossible in a P2P network and coordinating it between locations would be an even bigger nightmare. I imagine this is why turning on an entirely new network is a more viable option for them.
Re:Skype brief post-mortem by dennisgorelik on Wednesday December 22, 2010 @05:27PM (#2032704)
Why would overloaded server go down? Shouldn't it simply stop serving incoming requests if it's overloaded?
Re:Skype brief post-mortem by barrkel on Wednesday December 22, 2010 @05:32PM (#2032718)
Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?
Re:Skype brief post-mortem by dennisgorelik on Wednesday December 22, 2010 @05:38PM (#2032733)
There is a meaningful distinction between a server that serves 1% of requests and server that is down.

I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge.

Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node?

Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?

Re:Skype brief post-mortem by moe on Wednesday December 22, 2010 @06:15PM (#2032833)
I assume that overloaded server still serves some requests.

See http://en.wikipedia.org/wiki/Thundering_herd_problem

Many types of systems need a warm up period before they can realize their full performance. In web applications a controlled warm up is often needed to prime the caches. In P2P applications - which you can't easily "reboot" as a whole - the restoration of a steady state can be much more complex.

Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?

In theory, yes. In practice this seems to be a case of http://en.wikipedia.org/wiki/Cascading_failure

The remaining supernodes either can't handle the aggregate load alone. Or they are being overwhelmed because the re-connection attempts from clients are not evenly distributed.

Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?

In theory, yes. In practice there's probably a lot of http://en.wikipedia.org/wiki/Positive_feedback and perhaps even http://en.wikipedia.org/wiki/Monster_wave going on in the skype network right now.

Re:Skype brief post-mortem by jemfinch on Thursday December 23, 2010 @02:47AM (#2033841)
This has absolutely nothing to do with the thundering herd problem. Why are you linking to seemingly random Wikipedia articles?

The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.

Re:Skype brief post-mortem by lusis on Thursday December 23, 2010 @05:41AM (#2034082)
True, there's no reason but that doesn't mean it doesn't happen. Many people (I would argue, rightly) equate degraded service to being out of service.

Made-up Scenario: My cluster can handle XXXk users with an SLA of YYms response time. In degraded mode, I'm only handling XXk users with YYYYms response times.

I'm not meeting my SLA for the remaining number of users so I am, in essence, offline.

As to your specific point of "crashing", look at what happened with 37s. Should the server have crashed? No but there was a bug. The reason you add more capacity in the FIRST place is because the existing number of nodes cannot handle the volume. Depending on any number of bugs, issues or configuration your degraded capacity is for all intents and purposes "crashed".

Made-up scenario #2: A single server in your apache configuration can handle 200k concurrent connections reliably with fast response times. Double that load and response times are so long that various devices on the path are timing out the connections as stale. Apache hasn't crashed but it's not really doing anything.

Fast failure is an accepted best practice. Shit, it's baked into Erlang. Kill the process, start a new one and move on. Depending on the nature of the crash, you're doing nothing but churning processes and not actually servicing requests.

The bigger problem is that people don't design for this type of scenario. Static landing pages. Decoupled services instead of monolithic all-in-one containers. Look at github. That's an awesome example of how to degrade service during an outage. Only certain components are "crashed" because everything is fairly decoupled.

Meanwhile there's a guy over here running 4 apps in the same tomcat container that communicate over memory transport with each other or even if he had the common sense to decouple each app, didn't bother to fail fast and was busy spinning up threads trying to communicate with the rest of the services that he can't actually respond to anything externally.

Re:Skype brief post-mortem by jemfinch on Thursday December 23, 2010 @01:47PM (#2035407)
You're missing a fundamental difference between degraded service and being out of service: if an overloaded server handles its maximum number of clients but doesn't crash, then you have time to bring up additional servers to share the load. If an overloaded server crashes instead of limiting the number of clients it serves, you have to staunch the flow of clients further upstream before you can bring more servers up to share the load. This is what the GGGP is saying, and it seems like everyone (including you) are missing his point.

Degrading service, or more accurately, serving the maximum number of clients you can serve and no more is absolutely not the same thing as being down.

Failing/crashing rather than limiting the number of clients you serve is absolutely not established best practice. It's poor practice, in fact.

Re:Skype brief post-mortem by lusis on Monday December 27, 2010 @08:06AM (#2042408)
I'm approaching it from the business perspective. If sipport staff are getting calls for poor performance then the product isn't working. End of story. Business units don't care about shades of gray and degraded service. It's binary for them. It's either working or it isn't. Couple that with financial penalties for latency and degraded service means even less. Either you're meeting your SLA or you aren't.
Re:Skype brief post-mortem by moe on Thursday December 23, 2010 @07:28AM (#2034217)
This has absolutely nothing to do with the thundering herd problem.

The skype outage probably not. This was in response to the thread going on about overloaded webapps - thundering herd is a common issue there.

The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.

Nobody used the word "crash" but graceful degradation is quite hard. You have to understand the problems at heart, plan and build for it. That is more than enough reason for most apps to not degrade very gracefully.

Re:Skype brief post-mortem by jemfinch on Thursday December 23, 2010 @01:49PM (#2035424)
Thundering herd is an efficiency problem on certain implementations of UNIX servers. It's not a common reason why servers crash when overloaded.

Graceful degradation is not hard. Serve N clients, and no more. It's an extra conditional in the event loop. Am I serving N or more clients right now? Then don't accept new connections (or accept(2) them and immediately close them to avoid a backup in the listen queue).

Re:Skype brief post-mortem by moe on Thursday December 23, 2010 @05:29PM (#2036069)
Thundering herd is an efficiency problem on certain implementations of UNIX servers.

The accept()-issue you're referring to is merely a popular manifestation of the thundering herd problem.

The term is nowadays also used in a broader sense to describe similar situations in distributed systems.

It's not a common reason why servers crash when overloaded.

Again, nobody is talking about servers crashing here.

We are talking about web applications (which usually consist of multiple servers talking to each other) degrading to the point of not serving a meaningful rate of requests anymore.

In that context the role of the "thundering herd" is often taken by your own users, who will furiously smash their reload-buttons to get past those 500-errors.

It's an extra conditional in the event loop. Am I serving N or more clients right now?

You're talking about implementation details of a socket server. I was talking about entire applications.

Re:Skype brief post-mortem by barrkel on Wednesday December 22, 2010 @06:16PM (#2032841)
It usually doesn't work that way. Usually, when your server is overloaded, instead of serving some fraction of the requests in reasonable time, it tries to queue the requests and serve them in order, and as a result response time goes through the roof. Often, the user at the other end of the causal chain starts getting frustrated and refreshes, adding more requests to the queue.

When servers are getting overloaded, you need to throttle the incoming requests aggressively and early, to keep response time reasonable.

Re:Skype brief post-mortem by JoachimSchipper on Thursday December 23, 2010 @08:06AM (#2034294)
In network engineering, systems quite often use RED (http://en.wikipedia.org/wiki/Random_early_detection) - proactively drop a percentage of connections/packets/... when loads begins to climb, scaling from 0% (any easily handled load) to 100% (at capacity). In practice, this results in pretty robust systems. (But note that TCP et al. have mechanisms that help this, e.g. slowing down when packets are lost and retrying a connection with a backoff.)
Re:Skype brief post-mortem by terryjsmith on Wednesday December 22, 2010 @05:55PM (#2032770)
It handling 1% of connections would assume that the network was the bottleneck. You are much more likely to use up the CPU, RAM, or other resources before hitting the maximum number of available sockets. In which case things are swapping or waiting for available CPU time and each individual requests becomes seconds or tens of seconds to get handled. For all intents and purposes, that machine is dead.

As to your second point, I think you are right. I assume that is what the mega-supernode is: a network of machines who's resources are as high as can be to handle all of the connections and try to beat the bottlenecks.

Re:Skype brief post-mortem by dalore on Thursday December 23, 2010 @06:46AM (#2034164)
Usually too many requests mean the machine runs out of memory. How it handles that is different, but it usually means the service goes down and might need a restart.
Re:Skype brief post-mortem by tommi on Wednesday December 22, 2010 @05:13PM (#2032660)
That can be avoided by altering the logic by which supernodes are queried. Not all nodes must try all supernodes.
Re:Skype brief post-mortem by terryjsmith on Wednesday December 22, 2010 @05:17PM (#2032681)
Since it seems a good chunk of the super nodes went down, I imagine every running unconnected Skype instance is checking all of the super nodes it can find (or actively searching for them). Changing the logic at this point isn't really an option for them.
Re:Skype brief post-mortem by tommi on Wednesday December 22, 2010 @05:22PM (#2032693)
Well, even if we were to talk about this point, it actually might be. I have no knowledge of Skype architecture, so I'm totally guessing here. But, those unconnected Skype instances have to have some kind of directory of supernodes: either dynamic or static. If it's dynamic, that is from Skype's servers, it can be affected.
Re:Skype brief post-mortem by terryjsmith on Wednesday December 22, 2010 @05:30PM (#2032713)
That's fair, but even if it is dynamic, I doubt it's checked on a regular basis; I doubt it's a scenario they've even considered before now. This would be on par with the DNS root servers changing and somehow letting everyone know about it on the fly.
Re:Skype brief post-mortem by tommi on Wednesday December 22, 2010 @05:41PM (#2032736)
I tend to disagree. Error handling exactly in such cases where you have to bring the network online from a complete halt or other catastrophe is at the very heart of the architecture of these systems. These scenarios should and are at the minds of the architects and coders.
Re:Skype brief post-mortem by terryjsmith on Wednesday December 22, 2010 @05:46PM (#2032750)
That's a lot easier when it's your network. Skype is obviously designed to be used in "hostile" environments; super nodes are likely expected to go away and come back on a regular basis. Likewise, I don't think it's an unreasonable expectation that there will be a given number of nodes online at any given time.

If they all disappear, I doubt there's a contingency plan for that.

Re:Skype brief post-mortem by tommi on Wednesday December 22, 2010 @05:58PM (#2032777)
From the Skype's status update: "Our engineers are creating new ‘mega-supernodes’ as fast as they can, which should gradually return things to normal."

So they can inject new supernodes in to the system and clients will discover'em gradually. That kind of indicates that they have a mechanism not to DDOS the new mega-supernodes. It's a contingency plan. I fail to see what we are arguing here.

Re:Skype brief post-mortem by terryjsmith on Wednesday December 22, 2010 @06:04PM (#2032799)
I'm just relaying my experience with P2P network design and with system-wide failures. I agree with you that gradual discovery is possible, but I doubt that's the best case scenario for them depending on how long it takes. They have the worst of both world: if it's slow, their users are going to be extremely frustrated, but if their clients are very quick to discover new nodes, the possibility of them getting swamped is still very real. The status update doesn't talk about their deployment strategy at all so I don't think we can make assumptions; either way has it's own potentially serious issues.