News for hackers. Stuff that matters.
Re:Don't Use an RDBMS for Messaging by jacques_chester on Sunday September 28, 2014 @09:01AM (#8379610)
State machines and protocols are an excellent way to model the problem of managing the lifecycle of a single task or LRP; they don't solve the problem of the overall system. It's the latter which is actually much harder.

Consider the classic termite-and-wood simulation[0]. In this simulation I have a population of termites, randomly scattered on a grid. And flakes of wood, also randomly distributed. I want to gather this wood.

A current-generation approach would be to, for example, divide the grid into subgrids, assign local termites to perform exhaustive search, progressively aggregating into larger piles in successive rounds. This would require substantial engineering to ensure that my central wood-gathering algorithm has checkpointing, failover, robust message passing, reruns etc etc.

Alternatively, I can give each termite 3 simple rules:

1. Walk randomly.

2. If you don't have a piece of wood in your jaw and you come across one, pick it up.

3. If you have a piece of wood in your jaw and you come across another piece of wood, drop the wood in your jaw.

This latter solution doesn't have difficult -- in some cases impossible -- engineering requirements. And it works very well.

Most systems at this scale eventually wind up push intelligence out to the nodes, because (forgive my wankery here) it's impossible in a relativistic universe to have perfect simultaneous knowledge of phenomena separated by space and time. You need to subdivide the problem so that agents can solve their own and rely on emergent behaviour in the overall system.

[0] http://ccl.northwestern.edu/netlogo/models/Termites

Re:Don't Use an RDBMS for Messaging by Rapzid on Monday September 29, 2014 @02:17AM (#8381859)
This makes a lot of sense for certain problems in the domain. I've thought a few times about how to introduce the right bit of agency into actors in order to have them accomplish a larger goal without a central authority.

I created a couchbase cluster joiner that would have the nodes converge into a single initial cluster based on ec2 tags. I thought about how people would group up on their own when walking into a room and came up with:

  Join a group
  Judge value of group(is it the largest + tie breaker)
  Leave and join group of highest value
The problem wasn't so hard as the termites but it removed the need for a central controller.. And I was proud of myself :)