News for hackers. Stuff that matters.
Re:Monoliths Are the Future by segmondy on Thursday January 30, 2020 @03:15PM (#22195037)
no it's not.

ETL is really database focused and batch focused , Extract, Transform, Load.

Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/

Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capt...

Overview

"Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."

Re:Monoliths Are the Future by cleong on Thursday January 30, 2020 @04:39PM (#22196007)
What your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.