Re:Streamz: Python pipelines to manage continuous streams of...
by yingw787 on Thursday March 26, 2020 @09:14PM
(#22699656)
Very cool! I'm interested in learning more about the memory model of multi-stage streaming systems. From my limited understanding of distributed data systems, we went from saving files and checkpointing completely on disk a la MapReduce to a disk/memory hybrid with Spark RDDs and lazy job execution (not sure at all?) to pure streaming like Kafka. Could somebody please enlighten me? I'd love to learn more.
[ Parent
| on HN ]
Re:Streamz: Python pipelines to manage continuous streams of...
by sanderjd on Thursday March 26, 2020 @09:43PM
(#22699808)
I just started reading Streaming Systems: https://www.goodreads.com/book/show/34431414-streaming-syste.... Seems really great so far.
Re:Streamz: Python pipelines to manage continuous streams of...
by tarun_anand on Thursday March 26, 2020 @09:30PM
(#22699747)
I would add that Kafka is a distributed commit log. Streaming is one of the applications. Spark also supports streaming. One should distinguish between the compute model like MapReduce, RDD etc and Data Model like streaming.
HTH