News for hackers. Stuff that matters.
Re:Queues Don't Fix Overload by IgorPartola on Thursday November 20, 2014 @12:53AM (#8634599)
We don't have fully async computers. Something somewhere will block, be it in the CPU, fetching blocks from a disk, kernel context switches, etc. You will hit fundamental limits. Heck, I've hit network card limits before and was none the wiser. There is actually a really good question you can ask yourself when designing a system: do you want it to go very fast at times, or do you want it to have very predictable speed? I learned to choose the latter: better fire up 16 processes and delegate work to them from a master process than to fire up one async loop that may get bogged down at some point.

At the same time, having a good mental model of what your code is waiting on or is blocked by is very important. Years ago I read advice on building multi-threaded systems. It went something like "when asked what's the hardest thing about coding up multi-threaded systems, beginners say things like deadlocks, but real pros say 'avoiding doing nothing'". So if your application does some large bit of work and in the middle of it it's blocked, why not go do something else.

Here's the hitch, and what the article alludes to: a system where you have 16 worker processes and they are either handling a client request or are waiting, your max users is 16. That's it, it's a hard limit. By design you can't process 17 requests at once. If you however go with an async system, let's say based on Node.js, you don't have this control. You have a crude "how many connections can we accept" thing, but you generally can't introduce limits you can actually reason about. If you simply set your max connections wide open, and then connect it all to MongoDB, you'll quickly find that MongoDB has a global write lock that will start rearing its ugly head. You can't overcome this limit, but you also don't know where this limit is. Systems with set internal limits that propagate back pressure to the edge, on the other hand, have very predictable behavior.

Re:Queues Don't Fix Overload by Illniyar on Thursday November 20, 2014 @01:27AM (#8634650)
I think you are ignoring a few things about web development.

First, 16 workers don't equal 16 concurrent users if you have less then 16 cores in your server, if you don't then you still have a queue, it's just the queue the OS uses to handle context switches - which is even harder to reason about.

Second, the entire idea behind event-based processing (what node.js and Java based netty or vert.x) as opposed to thread-based is that the bottleneck is rarely the cpu. Event-based processing is one of the ways to "avoid doing nothing".

Re:Queues Don't Fix Overload by Denzel on Thursday November 20, 2014 @03:00AM (#8634773)
Minor nitpick: concurrency is not parallelism. 16 workers does equal 16 concurrent users.