- How do you version your messages?
- What problem domain are you working on?
- What kind of process orchestration system do you use?
Process orchestration: Currently very simple. (Python multiprocessing). May change to something more complex in future.
Message versioning: It depends. My configuration management / deployment model means that most of the time backwards compatibility is not required. (The entire system is deployed as a single unit). When messages are persisted, the data recording system files the messages in a configuration management system that knows the system configuration that generated those messages -- so that when they are replayed back -- e.g. to test a new configuration -- the appropriate reading/de-serialization logic can be selected.
I'm building something on similar lines with Redis and Protobuf, but I'm in over my head a little bit
It is worth noting: this is part of a custom code-generation/build/test/parameter-tuning system that I developed for embedded machine-vision/sensor-data-processing systems.
It is not intended to be used by third-parties; or distributed as a library -- it is very much integrated into my scripted workflow.
I declare my data structures in YAML. From this I generate code for memory allocation; data validation; serialization and de-serialisation routines.
I also declare the topology of the data flow graph in a separate YAML file, decorating each edge with the data type.
Finally, I have a separate YAML file that specifies which computers/processes each node will run on.
Nodes communicate by passing messages. For nodes in the same process, this is simply shared memory (zero overhead). For nodes in different processes that share a memory bus, this is a shared memory queue. For nodes on different machines, the data goes over the network.
Nodes can be python, C or C++.
Very nice. Very declarative. The bare minimum of duplication and boilerplate. Changing from MIL to SIL to HIL is (mostly) just a configuration change. Logging, measurement and replay is built into the framework, as is support for parameter tuning.
Within a single process, the model of computation is synchronous data flow. Across processes, it is asynchronous data flow, but because we have queues with blocking reads and nonblocking writes, we still keep the ability to replay data through the system in a deterministic manner. (Modulo nodes exceeding their time budget).