Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @11:12AM
(#35169543)
More of a higher level library like Beam, and I could see it being plugged into a Prefect workflow.
[ Parent
| on HN ]
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by jcnnghm on Wednesday March 15, 2023 @11:30AM
(#35169839)
Is there an underlying stream processor (e.g. Flink)? How many messages per second can it process?
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @11:34AM
(#35169899)
All of our processing is done via Ray (https://www.ray.io/). Our early benchmarks are about 5k mesesages per second on a single 4 core VM, but we believe we can increase the with some more optimizations.
This bench mark was consuming a Google Cloud Pub/Sub stream and outputting to BigQuery.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by dmatrixjsd on Wednesday March 15, 2023 @11:26PM
(#35177784)
Delighted to hear your choice of Ray and building atop Ray.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by faizshah on Wednesday March 15, 2023 @12:02PM
(#35170385)
I see, what fault tolerance mechanisms does it provide?
I don’t see anything on snapshotting or checkpointing like Flink. Is this just for stateless jobs?
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @12:10PM
(#35170515)
We don't support any snapshotting or checkpointing directly in BuildFlow at the moment, but these are great features we should support.
But we do have some fault tolerance baked into our I/O operations. Specifically for Google Cloud Pub/Sub the acks don't happen until the data has been successfully processed and written to the sink, so if there is a bug or some transient failure the message will be resent later depending on your subscriber configuration.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @04:28PM
(#35174276)
I should also mention BuildFlow does support stateful processing with the Processor class API: https://www.buildflow.dev/docs/processors/overview#processor...