Re:Mara: A lightweight ETL framework, halfway between plain ...
by bduerst on Wednesday May 09, 2018 @11:58AM
(#17030853)
Some ETLs these days are simple scripts - e.g. Spark, Dataflow - with their config files.
Re:Mara: A lightweight ETL framework, halfway between plain ...
by cosmie on Wednesday May 09, 2018 @12:28PM
(#17031174)
Being able to define an ETL workload within a simple script is not the same as your ETL system itself being a simple script.
While I love both Spark and Dataflow, both of them are incredibly complex distributed systems with very high operational costs. Someone, somewhere is paying a lot of money to have an operational resource maintain that complexity. Whether you have an internal devops resource doing so or you're using a managed service, you're paying for that complexity somehow. And, for a lot of workloads, you aren't actually getting any more value than you would from standing up a ~$50/month standard Debian/Ubuntu server and a set of simple scripts on it.
+1
> Unless you need multiple servers processing data through multiple steps and you need the auditing and process control
I'll stress the "multiple servers" part. You can add in a substantial amount of multiple, sequential steps and auditing and process control in a simple script. The part that adds orders of magnitude worth of complexity and operational overhead and points of failure is being able to distribute it to multiple servers. Distributed architectures are operationally and architecturally expensive. And far more often than not, completely unnecessary for a given use case.