Re:Streamz: Python pipelines to manage continuous streams of...
by mrlucax on Thursday March 26, 2020 @08:49PM
(#22699528)
Does anyone know of any good resources to learn about data streams in general? Some weeks ago I had to implemnt some streams (in nodejs) to upload a file to s3 "on the fly", without storing the file locally and then uploading it, but I culdn't wrap my head about the data stream concept.
[ Parent
| on HN ]
Re:Streamz: Python pipelines to manage continuous streams of...
by yingw787 on Thursday March 26, 2020 @09:22PM
(#22699705)
Have you reviewed data-intensive architectures? I've found that book quite useful.
I'm just talking out my butt right now but I think fundamentally, a stream is just a chunk of data lifted from persist into memory. I imagine a cursor process traversing some bytes in a file, and then lifting some of those bytes into memory, and sending that memory over network.
Re:Streamz: Python pipelines to manage continuous streams of...
by alexpetralia on Thursday March 26, 2020 @09:32PM
(#22699754)
Yes, however I think there's a lot more complexity in modern steaming architectures: event sourcing, concurrency, eventual consistency, pub-sub, queueing, event handlers, microservices, stateless messages, data lakes - to name a few. I would also be interested in a resource that tackles major concepts here.
Re:Streamz: Python pipelines to manage continuous streams of...
by CameronNemo on Thursday March 26, 2020 @09:36PM
(#22699775)
> data-intensive architectures
I suspect you mean __Designing Data-Intensive Applications__, by Martin Kleppman, but I am not entirely sure.
Re:Streamz: Python pipelines to manage continuous streams of...
by yingw787 on Thursday March 26, 2020 @09:57PM
(#22699882)
Yeah, I just didn't want to type it out :P
Re:Streamz: Python pipelines to manage continuous streams of...
by dttos on Friday March 27, 2020 @02:08AM
(#22700733)
Check out dataflows in Composable for a quick way to implement these data streams.
https://docs.composable.ai/en/latest/03.DataFlows/01.Overvie...
Re:Streamz: Python pipelines to manage continuous streams of...
by toolslive on Friday March 27, 2020 @05:35AM
(#22701423)
you can also use the upload multipart feature of s3. Basically, you buffer until you have a sizeable chunk of the incoming stream, then upload that chunk; iterate. At the end you tell s3 to concatenate the parts.