Re:The unintentional side-effects of a bad concurrency model
by toast0 on Wednesday January 27, 2016 @03:11AM
(#10978876)
> How is it easier to debug a message-passing architecture than a monolith?
You can observe the messages as they're passed, which is usually enough. In my experience in Erlang, a lot of messages include a pid to reply to, so if you get an unexpected message, you can often see where it came from, and then look at the messages that came into that process, etc (or grep the code on likely identifiers). Erlang makes this pretty easy, you can also peek at the mailbox for a process to see what other messages it hasn't read yet, etc. This model works really well for me, because I've been using it for years to debug networking applications with tcpdump and friends: figure out what's gone wrong over the wire, then you can figure out where it's gone wrong in the code.
Added: I guess that doesn't really address emergent behavior. Most of the emergent behavior I've seen is really around overload conditions, most of the debugging around that is focused around finding the bottleneck(s) (which is usually a problem in a simple process -- you can tell which ones because they have a huge mailbox), why there isn't enough back pressure (it's usually because the senders aren't waiting for confirmation, or aren't waiting long enough, especially in combination with retries), and how to shed load instead of falling over (which is tricky)
You can observe the messages as they're passed, which is usually enough. In my experience in Erlang, a lot of messages include a pid to reply to, so if you get an unexpected message, you can often see where it came from, and then look at the messages that came into that process, etc (or grep the code on likely identifiers). Erlang makes this pretty easy, you can also peek at the mailbox for a process to see what other messages it hasn't read yet, etc. This model works really well for me, because I've been using it for years to debug networking applications with tcpdump and friends: figure out what's gone wrong over the wire, then you can figure out where it's gone wrong in the code.
Added: I guess that doesn't really address emergent behavior. Most of the emergent behavior I've seen is really around overload conditions, most of the debugging around that is focused around finding the bottleneck(s) (which is usually a problem in a simple process -- you can tell which ones because they have a huge mailbox), why there isn't enough back pressure (it's usually because the senders aren't waiting for confirmation, or aren't waiting long enough, especially in combination with retries), and how to shed load instead of falling over (which is tricky)