News for hackers. Stuff that matters.
Re:Streamz: Python pipelines to manage continuous streams of... by yingw787 on Thursday March 26, 2020 @09:22PM (#22699705)
Have you reviewed data-intensive architectures? I've found that book quite useful.

I'm just talking out my butt right now but I think fundamentally, a stream is just a chunk of data lifted from persist into memory. I imagine a cursor process traversing some bytes in a file, and then lifting some of those bytes into memory, and sending that memory over network.

Re:Streamz: Python pipelines to manage continuous streams of... by alexpetralia on Thursday March 26, 2020 @09:32PM (#22699754)
Yes, however I think there's a lot more complexity in modern steaming architectures: event sourcing, concurrency, eventual consistency, pub-sub, queueing, event handlers, microservices, stateless messages, data lakes - to name a few. I would also be interested in a resource that tackles major concepts here.
Re:Streamz: Python pipelines to manage continuous streams of... by CameronNemo on Thursday March 26, 2020 @09:36PM (#22699775)
> data-intensive architectures

I suspect you mean __Designing Data-Intensive Applications__, by Martin Kleppman, but I am not entirely sure.

Re:Streamz: Python pipelines to manage continuous streams of... by yingw787 on Thursday March 26, 2020 @09:57PM (#22699882)
Yeah, I just didn't want to type it out :P