News for hackers. Stuff that matters.
Re:Mara: A lightweight ETL framework, halfway between plain ... by pgwhalen on Wednesday May 09, 2018 @12:20PM (#17031072)
Is anyone out there doing (or considering) streaming ETL as opposed to batch?
Re:Mara: A lightweight ETL framework, halfway between plain ... by woqe on Wednesday May 09, 2018 @12:36PM (#17031265)
Apache nifi (https://nifi.apache.org/) is a software which handles streaming ETL processing.

From a previous post I made about it: "We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints."

Re:Mara: A lightweight ETL framework, halfway between plain ... by chrisjc on Wednesday May 09, 2018 @12:40PM (#17031308)
There's often a misconception about Nifi being batch or file based and not streaming. This probably originates as result of data in Nifi being represented as a FlowFile, which is kind of a misnomer. As woqe stated above, Nifi can certainly doing streaming.
Re:Mara: A lightweight ETL framework, halfway between plain ... by artwr on Wednesday May 09, 2018 @12:24PM (#17031126)
I think a lot of people are considering it. The ideas behind Apache Beam are supposed to make the transition easier, with the ability to run in either mode. I think in practice it really depends on your use case. For a large enough amount of data, especially on the ingest side, running it streaming makes sense. Now for that daily report from aggregated table, do you really need streaming ETL/aggregation is another question. The operational cost is non trivial with streaming.
Re:Mara: A lightweight ETL framework, halfway between plain ... by pgwhalen on Wednesday May 09, 2018 @01:46PM (#17031994)
Can you expand on that last sentence? Not that I’d disagree, I’m just curious about what specifically you’re referring to.

My team is currently underwater supporting a decades old system of batch ETL, where it’s a major challenge to understand dependencies between jobs, and therefore what to rerun when something fails. We’re exploring a move to streaming ETL (Kafka streams being the leading candidate) as a way of making the dependencies explicit: rather than having to rerun stored procs or Talend jobs that we depend on certain data, we run a fleet of 24/7 microservices that transform and move data downstream whenever it is available.

Re:Mara: A lightweight ETL framework, halfway between plain ... by thibaut_barrere on Wednesday May 09, 2018 @02:29PM (#17032465)
I do both depending on the project needs!