News for hackers. Stuff that matters.
Re:Mara: A lightweight ETL framework, halfway between plain ... by cosmie on Wednesday May 09, 2018 @11:25AM (#17030535)
> you probably don’t need an ETL, just a simple script.

+1

> Unless you need multiple servers processing data through multiple steps and you need the auditing and process control

I'll stress the "multiple servers" part. You can add in a substantial amount of multiple, sequential steps and auditing and process control in a simple script. The part that adds orders of magnitude worth of complexity and operational overhead and points of failure is being able to distribute it to multiple servers. Distributed architectures are operationally and architecturally expensive. And far more often than not, completely unnecessary for a given use case.

Re:Mara: A lightweight ETL framework, halfway between plain ... by bduerst on Wednesday May 09, 2018 @11:58AM (#17030853)
Some ETLs these days are simple scripts - e.g. Spark, Dataflow - with their config files.
Re:Mara: A lightweight ETL framework, halfway between plain ... by cosmie on Wednesday May 09, 2018 @12:28PM (#17031174)
Being able to define an ETL workload within a simple script is not the same as your ETL system itself being a simple script.

While I love both Spark and Dataflow, both of them are incredibly complex distributed systems with very high operational costs. Someone, somewhere is paying a lot of money to have an operational resource maintain that complexity. Whether you have an internal devops resource doing so or you're using a managed service, you're paying for that complexity somehow. And, for a lot of workloads, you aren't actually getting any more value than you would from standing up a ~$50/month standard Debian/Ubuntu server and a set of simple scripts on it.

Re:Mara: A lightweight ETL framework, halfway between plain ... by Mironor on Thursday May 10, 2018 @08:55AM (#17038402)
You don't have to have a cluster to run spark scripts, setting master to `local` (and running it on one machine) is often enough for small anounts of data.
Re:Mara: A lightweight ETL framework, halfway between plain ... by bduerst on Thursday May 10, 2018 @12:39PM (#17040479)
They don't have high operational costs - you can run them as a script on your local machine. You're making them out to be more complex than they really are.