Since Node bounces much work to the kernel then kernel and TCP factors can play a part. So ulimit, and ephemeral port tuning is important.
Finally at some point CPU load for doing keep-alive on open lots of connections has a cost. So that is also a limiting factor.
In general it depends what you are doing. Using Joyent's SmartOS (a Solaris varient) I was able to hold just under a million TCP connections on a single Node process. However the ability to service those Node is extremely limited.
Would every request still stay in a queue, not on the front-end layer, but on database/storage/messaging/other middleware layer?
I am just wondering what are the benefits of moving a bottleneck one layer up?
My point is more that Node can support a very large amount of connections. Once you scale down to the tens or possibly low hundreds of thousands you can still do real work.
This is mostly useful when you have a server that is looking after a lot of things that don't communicate very often. Sensors is one application for example. While totally passive sensors can poll a server with their data, being able to hold open many very cheap connections allows you ask sensors for their data when the server wants it rather than on an interval. This can cut down the amount of unnecessary data sent a lot.
I've read similar sentiments in a bunch of places but whenever I push any real traffic at the nodejs version of our API server it falls over pretty quickly (even with ulimit increases). What exactly are people doing to handle tens or hundreds of thousands of connections?
Incidentally, if any nodejs gurus are interested we're looking for a remote freelancer to audit our nodejs stuff and help us with a deployment strategy (assuming we can get to the 100s of thousands of connections mark).