News for hackers. Stuff that matters.
Re:Meltano: ELT for the DataOps era by kbd on Monday March 01, 2021 @10:48AM (#26303895)
This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?
Re:Meltano: ELT for the DataOps era by JPKab on Monday March 01, 2021 @11:30AM (#26304445)
While there are technical differences with transforming the data within it's storage location/database, I've interacted with my former employer's board of directors enough to know there's a bigger, rather obnoxious reason for this:

VC valuations in Silicon Valley for traditional "ETL" tools isn't great. The term "ETL" has baggage of Talend and other big enterprise tools that have an aura of "not being user friendly".

"ELT" is intended to signal that a tool is trying really hard to not be like those sad, difficult to use, dirty "ETL" tools.

That's the real reason. There really isn't an actual difference in how the tools work. Traditional ETL tools are perfectly capable of loading the data and initiating the transform in the database it's being stored in. This has been done for decades.

I have been repeatedly lectured by senior strategy team executives about using the proper vernacular to ensure perception of value is maximized in the trendy, buzzword driven VC world. So fucking stupid.

Re:Meltano: ELT for the DataOps era by visch on Monday March 01, 2021 @10:53AM (#26303960)
"Transform" step is happening after the load to data warehouse here. https://blog.getdbt.com/what--exactly--is-dbt-/ is a pretty good overview.
Re:Meltano: ELT for the DataOps era by snidane on Monday March 01, 2021 @04:14PM (#26308503)
ELT means: let's focus on moving data between from system A to system B and skip the transform part, which is a business of whoever has access to the target system. Which is usually a sql database so you can then do arbitrary data transforms there.

In other words just get the data in house into a queryable form and worry about the T part later.

Re:Meltano: ELT for the DataOps era by nojito on Monday March 01, 2021 @12:18PM (#26305206)
Marketing. There's been a huge marketing push of tools that designed to automate the SQL portion of data pipeline.

Doesn't make much sense to me, because anyone who's been in the data space quickly sees the rampant marketing of different tools that all do the same thing but simply change the order of the pipeline.

Re:Meltano: ELT for the DataOps era by kpierce on Monday March 01, 2021 @01:23PM (#26306084)
Keeping a raw version of the data make it easier to change in the future. If you use s3 as a data lake you can transform later into the shape you want it. Also can use tools like presto where schema is on read instead of hardcoded so transform step is moved to another time.
Re:Meltano: ELT for the DataOps era by tayloramurphy on Monday March 01, 2021 @10:55AM (#26303972)
This article[0] from Census is a good primer on the difference and what's driving it. But the TL;DR is that storage is cheap and it's easier to change SQL-based transformations using something like dbt[1] than to re-extract the data when the business logic changes.

[0] https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ [1] https://www.getdbt.com/