Re:MonolithFirst
by vidarh on Wednesday June 03, 2015 @08:05PM
(#9656706)
One of my pet peeves is that developers rarely understand the costs of traversing boundaries well enough to even think about them.
You won't do distributed services well if you don't understand that each step removed from in-process, in-cache carries with it a severe cost-multiplier for communicating state.
E.g. most developers (rightly) assume that function/method calls are usually quite cheap, but we don't tend to be good at thinking about cache effects. So we lose the first big factor of performance by e.g. chasing pointers around.
Then people tend to (wrongly) assume that if function calls are cheap, this extends to system calls. Another multiplier, as people e.g. does tons of small read()'s instead of larger reads into in-process user-space buffers.
But I see people go further, and assume that database queries or http requests are cheap, often helped by having them hidden behind ORMs and various APIs that obscures how often they're making these queries even if they're logically aware they're calling them.
The end result is that often these applications get written with the assumption that a function call is a function call is a function call, without thought to whether or not that function call is just a handful of machine instructions, or will trash the cache, or traverses a process boundary of hundreds of cycles before anything gets executed, or executes a remote API that may take millions of cycles to return.
And then something is "slow" and the developer in question often have only a very vague idea what is likely to be slow, and stumbles around trying to measure various parts of it without thinking about the bigger picture of what information traverses what boundaries how frequently.
The same concern applies to reliability tradeoffs as well as performance (e.g. lack of understanding of implications for high-availability for example).
I think a huge proportion of problems making effective use of distributed systems boils down to this. There are hard challenges, sure, but if you are aware that each step across a boundary carries a certain penalty, you're already well on the way to actually thinking about why you want to move something across a boundary, and think about whether or not you understand the tradeoff well enough, and to structure things to e.g. minimize the number of boundary traversals (once you think of system calls as "slow" you are much more likely to entertain the idea of the slight extra complexity of ensuring you read/write as much as possible in one go, for example).
Even just teaching younger developers to be aware of this, and how to actually measure and profile more, would help immensely, I think (one of my favourite things to show people is just very basic crude use of "strace"; suddenly it becomes very obvious to a lot of people that some of the things they do are immensely inefficient). Not because it would teach them to do distributed systems right, but because it would give them the tools to recognise much earlier when they're headed into territory they don't know well enough yet, instead of finding out after they've already painted themselves into a corner.
You won't do distributed services well if you don't understand that each step removed from in-process, in-cache carries with it a severe cost-multiplier for communicating state.
E.g. most developers (rightly) assume that function/method calls are usually quite cheap, but we don't tend to be good at thinking about cache effects. So we lose the first big factor of performance by e.g. chasing pointers around.
Then people tend to (wrongly) assume that if function calls are cheap, this extends to system calls. Another multiplier, as people e.g. does tons of small read()'s instead of larger reads into in-process user-space buffers.
But I see people go further, and assume that database queries or http requests are cheap, often helped by having them hidden behind ORMs and various APIs that obscures how often they're making these queries even if they're logically aware they're calling them.
The end result is that often these applications get written with the assumption that a function call is a function call is a function call, without thought to whether or not that function call is just a handful of machine instructions, or will trash the cache, or traverses a process boundary of hundreds of cycles before anything gets executed, or executes a remote API that may take millions of cycles to return.
And then something is "slow" and the developer in question often have only a very vague idea what is likely to be slow, and stumbles around trying to measure various parts of it without thinking about the bigger picture of what information traverses what boundaries how frequently.
The same concern applies to reliability tradeoffs as well as performance (e.g. lack of understanding of implications for high-availability for example).
I think a huge proportion of problems making effective use of distributed systems boils down to this. There are hard challenges, sure, but if you are aware that each step across a boundary carries a certain penalty, you're already well on the way to actually thinking about why you want to move something across a boundary, and think about whether or not you understand the tradeoff well enough, and to structure things to e.g. minimize the number of boundary traversals (once you think of system calls as "slow" you are much more likely to entertain the idea of the slight extra complexity of ensuring you read/write as much as possible in one go, for example).
Even just teaching younger developers to be aware of this, and how to actually measure and profile more, would help immensely, I think (one of my favourite things to show people is just very basic crude use of "strace"; suddenly it becomes very obvious to a lot of people that some of the things they do are immensely inefficient). Not because it would teach them to do distributed systems right, but because it would give them the tools to recognise much earlier when they're headed into territory they don't know well enough yet, instead of finding out after they've already painted themselves into a corner.