News for hackers. Stuff that matters.
Re:Rethinking Data Ingestion as a DAG by quinndupies on Tuesday February 17, 2026 @03:31PM (#47052893)
Once you decomposed the pipeline into stages, did observability become significantly easier? I imagine stage-level latency and queue depth give you a much clearer signal than profiling a single async service.
Re:Rethinking Data Ingestion as a DAG by apoorvas20 on Tuesday February 17, 2026 @04:57PM (#47053996)
Exactly. Once each stage in the pipeline lived as its own job in the queue, observability got dramatically better. Stage-level latency, queue depth, and failure rates made bottlenecks obvious in a way that was hard to see inside a single async service. It also made it much easier to tune concurrency per stage instead of guessing globally.