Re:Monoliths Are the Future
by derision on Thursday January 30, 2020 @02:05PM
(#22194290)
"data pipeline" is just the new trendy phrase for ETL, which the GP mentioned. just because a solution exists, does not make it a solved problem. it's not the right solution for everyone
[ Parent
| on HN ]
no it's not.
ETL is really database focused and batch focused , Extract, Transform, Load.
Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/
Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capt...
Overview
"Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."