News for hackers. Stuff that matters.
Re:Faust: Stream Processing for Python by nemothekid on Tuesday July 31, 2018 @07:00PM (#17658069)
The docs link doesn't seem to be live yet:

http://docs.fauststream.com/en/latest/introduction.html

One question I had is how Robinhood might be doing message passing and persistence. For example, I might have a stream, that creates more messages that needs to processed, and then eventually I would want to persist the "Table" to an actual datastore.

Re:Faust: Stream Processing for Python by asksol on Tuesday July 31, 2018 @07:26PM (#17658225)
My name is Ask and I am one of the co-creators along with Vineet.

Thanks for pointing this out. Fixed the links. You can also find the docs here: http://faust.readthedocs.io/en/latest/

Faust uses Kafka for message passing. The new messages you create can be pushed to a new topic and you could have another agent consuming from this new topic. Check out the word count example here: https://github.com/robinhood/faust/blob/9fc9af9f213b75159a54...

Also note that the Table is persisted in a log compacted Kafka topic. This means, we are able to recover the state of the table in the case of a failure. However, you can always write to any other datastore while processing a stream within an agent. We do have some services that process streams and storing state in Redis and Postgres.

Re:Faust: Stream Processing for Python by nemothekid on Tuesday July 31, 2018 @08:20PM (#17658545)
What I'm thinking of is flushing the table to a secondary storage so that other services can query that data.

I think Storm/Flink have the concept of a "tick tuple", a message that comes every n-seconds to tell the worker to flush the table to some other store. I've been looking over the project, and I'm not sure how I would do this in Faust yet, as far as I understand the "Table" is partitioned, so you'd have to send a tick to every worker.

Very interesting project!

Re:Faust: Stream Processing for Python by asksol on Tuesday July 31, 2018 @08:34PM (#17658615)
Faust can serve the data over HTTP/websockets and other transports, so you can query it directly!
Re:Faust: Stream Processing for Python by vineetgoel1992 on Tuesday July 31, 2018 @11:27PM (#17659421)
In addition to that you can have an arbitrary method run at an interval using the app.timer decorator. You can use this to flush every n seconds. You could also use stream.take to read batches of messages (or wait t seconds, whichever happens first) and write to an external source.