Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by jcnnghm on Wednesday March 15, 2023 @11:30AM
(#35169839)
Is there an underlying stream processor (e.g. Flink)? How many messages per second can it process?
[ Parent
| on HN ]
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @11:34AM
(#35169899)
All of our processing is done via Ray (https://www.ray.io/). Our early benchmarks are about 5k mesesages per second on a single 4 core VM, but we believe we can increase the with some more optimizations.
This bench mark was consuming a Google Cloud Pub/Sub stream and outputting to BigQuery.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by dmatrixjsd on Wednesday March 15, 2023 @11:26PM
(#35177784)
Delighted to hear your choice of Ray and building atop Ray.