News for hackers. Stuff that matters.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by calebtv on Wednesday March 15, 2023 @11:12AM (#35169543)
More of a higher level library like Beam, and I could see it being plugged into a Prefect workflow.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by jcnnghm on Wednesday March 15, 2023 @11:30AM (#35169839)
Is there an underlying stream processor (e.g. Flink)? How many messages per second can it process?
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by calebtv on Wednesday March 15, 2023 @11:34AM (#35169899)
All of our processing is done via Ray (https://www.ray.io/). Our early benchmarks are about 5k mesesages per second on a single 4 core VM, but we believe we can increase the with some more optimizations.

This bench mark was consuming a Google Cloud Pub/Sub stream and outputting to BigQuery.

Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by dmatrixjsd on Wednesday March 15, 2023 @11:26PM (#35177784)
Delighted to hear your choice of Ray and building atop Ray.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by faizshah on Wednesday March 15, 2023 @12:02PM (#35170385)
I see, what fault tolerance mechanisms does it provide?

I don’t see anything on snapshotting or checkpointing like Flink. Is this just for stateless jobs?

Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by calebtv on Wednesday March 15, 2023 @12:10PM (#35170515)
We don't support any snapshotting or checkpointing directly in BuildFlow at the moment, but these are great features we should support.

But we do have some fault tolerance baked into our I/O operations. Specifically for Google Cloud Pub/Sub the acks don't happen until the data has been successfully processed and written to the sink, so if there is a bug or some transient failure the message will be resent later depending on your subscriber configuration.

Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by calebtv on Wednesday March 15, 2023 @04:28PM (#35174276)
I should also mention BuildFlow does support stateful processing with the Processor class API: https://www.buildflow.dev/docs/processors/overview#processor...