chinmaychandak
Re:Streamz: Python pipelines to manage continuous streams of data
on Thursday March 26, 2020 @11:44PM (#22700286)
We at NVIDIA are heavily invested in developing streamz. We’ve made streamz compatible with RAPIDS cuDF, which means streaming jobs in Python can be GPU-accelerated now.
Folks interested in GPU-streaming data using 100% native Python (no Spark setup needed is a big win) can look up the Anaconda package called custreamz, which is part of NVIDIA RAPIDS open-source GPU Data Science libraries.
Streaming feature-parity-wise, we’ve made Kafka integration robust, and added checkpointing to streamz which is a must-have feature for production streaming pipelines.
I’d be happy to answer any questions you guys may have, and would love to have more people use streamz and contribute if possible.
As for scaling for big data streaming, streamz works well with Dask, so GPU-accelerated streaming in distributed mode is on! :)