Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @12:10PM
(#35170515)
We don't support any snapshotting or checkpointing directly in BuildFlow at the moment, but these are great features we should support.
But we do have some fault tolerance baked into our I/O operations. Specifically for Google Cloud Pub/Sub the acks don't happen until the data has been successfully processed and written to the sink, so if there is a bug or some transient failure the message will be resent later depending on your subscriber configuration.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel...
by calebtv on Wednesday March 15, 2023 @04:28PM
(#35174276)
I should also mention BuildFlow does support stateful processing with the Processor class API: https://www.buildflow.dev/docs/processors/overview#processor...
I don’t see anything on snapshotting or checkpointing like Flink. Is this just for stateless jobs?