Would every request still stay in a queue, not on the front-end layer, but on database/storage/messaging/other middleware layer?
I am just wondering what are the benefits of moving a bottleneck one layer up?
My point is more that Node can support a very large amount of connections. Once you scale down to the tens or possibly low hundreds of thousands you can still do real work.
This is mostly useful when you have a server that is looking after a lot of things that don't communicate very often. Sensors is one application for example. While totally passive sensors can poll a server with their data, being able to hold open many very cheap connections allows you ask sensors for their data when the server wants it rather than on an interval. This can cut down the amount of unnecessary data sent a lot.
Since Node bounces much work to the kernel then kernel and TCP factors can play a part. So ulimit, and ephemeral port tuning is important.
Finally at some point CPU load for doing keep-alive on open lots of connections has a cost. So that is also a limiting factor.
In general it depends what you are doing. Using Joyent's SmartOS (a Solaris varient) I was able to hold just under a million TCP connections on a single Node process. However the ability to service those Node is extremely limited.