News for hackers. Stuff that matters.
Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by faizshah on Wednesday March 15, 2023 @12:02PM (#35170385)
I see, what fault tolerance mechanisms does it provide?

I don’t see anything on snapshotting or checkpointing like Flink. Is this just for stateless jobs?

Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by calebtv on Wednesday March 15, 2023 @12:10PM (#35170515)
We don't support any snapshotting or checkpointing directly in BuildFlow at the moment, but these are great features we should support.

But we do have some fault tolerance baked into our I/O operations. Specifically for Google Cloud Pub/Sub the acks don't happen until the data has been successfully processed and written to the sink, so if there is a bug or some transient failure the message will be resent later depending on your subscriber configuration.

Re:Launch HN: BuildFlow (YC W23) – The FastAPI of data pipel... by calebtv on Wednesday March 15, 2023 @04:28PM (#35174276)
I should also mention BuildFlow does support stateful processing with the Processor class API: https://www.buildflow.dev/docs/processors/overview#processor...