vineetgoel1992

In addition to that you can have an arbitrary method run at an interval using the app.timer decorator. You can use this to flush every n seconds. You could also use stream.take to read batches of messages (or wait t seconds, whichever happens first) and write to an external source.

Not having to use systems such as Yarn/Mesos was a big motivation for us. Faust works well with any existing deployment systems (we currently use supervisor with EC2 instances) you may have and definitely made it easy for teams to start building/deploying apps. Further since Faust is a library, it can work with any existing python projects/tools - we use it both independently and alongside Django.

We have been using RocksDB for some time and it works fairly well. While we haven't seen any corruption yet, as you correctly pointed out, it is definitely a possibility.

We use RocksDB only as a cache and use log-compacted kafka topics as the source of truth. In the case of RocksDB corruption, we can simply delete the local rocksdb cache and the faust app should recover (and rebuild the rocksdb state) from the log compacted Kafka topic.

Hi, thanks for the comments. My name is Vineet and I am a co-creator of this project along with Ask.

We do plan on adding support for Kafka exactly once semantics to Faust. As @dleclere pointed out, streaming apps are simply a topology of consumers and producers. We will be adding this as soon as the underlying kafka clients add support for exactly once semantics.