News for hackers. Stuff that matters.
Re:Mara: A lightweight ETL framework, halfway between plain ... by pgwhalen on Wednesday May 09, 2018 @01:46PM (#17031994)
Can you expand on that last sentence? Not that I’d disagree, I’m just curious about what specifically you’re referring to.

My team is currently underwater supporting a decades old system of batch ETL, where it’s a major challenge to understand dependencies between jobs, and therefore what to rerun when something fails. We’re exploring a move to streaming ETL (Kafka streams being the leading candidate) as a way of making the dependencies explicit: rather than having to rerun stored procs or Talend jobs that we depend on certain data, we run a fleet of 24/7 microservices that transform and move data downstream whenever it is available.

There are no comments on this comment yet.