News for hackers. Stuff that matters.
Re:Notes on Distributed Systems for Young Bloods by rdtsc on Monday January 14, 2013 @11:15PM (#5058591)
> Implement backpressure throughout your system.

This is perhaps a point one cannot get to theoretically just sort of thinking about. This realization comes after observing the effects in practice. It is interesting to observe distributed systems, especially the ones that rely on asynchronous messages, thing go wrong and queues start filling up. Or even weirder machines start to synchronize for some reason. Like the size of the queues will oscillate in resonance of some sorts.

One way out is to reduce asynchronicity but that comes with serious performance penalties.

Re:Notes on Distributed Systems for Young Bloods by benjaminwootton on Tuesday January 15, 2013 @07:11AM (#5059740)
I view this the opposite way around. Message queues and asynchronicity are an alternative to backpressure as described.

If I synchronously hit some service then I need to build facilities into the service to say that it's too heavily loaded, and perhaps handle those responses on the client.

If I pop some job onto a message queue or asynchronous endpoint, things will just slow down for a while. Which is normally as good a way as any as soaking up load.

Re:Notes on Distributed Systems for Young Bloods by rdtsc on Tuesday January 15, 2013 @09:45AM (#5060306)
As someone already replied, message queues under constant load can grow unbounded.

> If I synchronously hit some service then I need to build facilities into the service to say that it's too heavily loaded, and perhaps handle those responses on the client.

Most of the time, you have to have timeouts when there is a synchronous API across the network.

In a large system ideally you'd want to reflect the level of loading back to the input source so the source can slow down sending the data. Think of TCP, tcp works this way on a small scale. One way to fix the problem is to leverage that to actually open a TCP stream and send data that way. The sender will slow down accordingly.

Now if you know that your load is not constant, so there are periods of high activity when inputs are generated then you can try and absorb (and amortize) those high volume peaks using asynchronous queues.

Re:Notes on Distributed Systems for Young Bloods by squarecog on Tuesday January 15, 2013 @07:36AM (#5059784)
How do you keep the message queue from unbounded growth?
Re:Notes on Distributed Systems for Young Bloods by jreichhold on Tuesday January 15, 2013 @09:29PM (#5064802)
Arrival rate and dequeue/processing time are the keys here. Unbounded queues are easy.. Just limit the queue size. Of course this isn't what the user of the service wants and leads to a very bad experience