Wanted to help answer your question as to what "optional normalized schemas" means. When writing data into your data warehouse we provide 2 options: 1. write each record as a json blob. 2. infer the schema of the data and write each value in a record to its own column with an appropriate type.
We are betting on EL(T), meaning we think that Transform should be considered separately from EL. To give a more a concrete example, if you are already using DBT in your data warehouse to normalize your data, you likely prefer operating on the "raw" (json blob) data than an arbitrarily normalized form of your data that your EL pipeline has decided on for you. I am seeing this trend pretty pervasively where a lot of, nominally, ELT pipelines are outsourcing the Transform to best in breed tools like DBT.
Thanks for asking this question btw, we'll do our best to clarify in our docs!
I'm skeptical of DBT's inference of dependencies between queries, but I'll keep an open mind until I have direct experience.
Happy hunting.
Or what I like to call 'data processing'.
That's more or less what I built for healthcare IT. Before the term "serverless" was coined.
Just peeked at Airbyte. I'd definitely look longer next time I have to do ETL.
--
My only pause for concern is their phrase "optional normalized schemas". What does that mean?
In my experience, data processing is best treated like screen or web scrapping. The tools based on schemas, mappings, patch cords (visual programing) always prove useless. All the CASE and workflowy tools like BizTalk and Talend are 800lb angry gorillas siting between you and your work.
Super simple self contained scripts, which can easily be ran from the command line or REPL, are The Correct Answer™.