Re:Airflow and the Future of Data Engineering: A Q&A
by vira on Wednesday March 01, 2017 @12:19PM
(#13765791)
Strong typing helps. Keep in mind enterprise ETL tools are designed to move data from oltp to olap databases, which are often strongly typed as well.
[ Parent
| on HN ]
Re:Airflow and the Future of Data Engineering: A Q&A
by gregn610 on Wednesday March 01, 2017 @01:03PM
(#13766389)
When a tool insist on you cast char fields to varchars before you can test two fields for equality, or keeps changing all your decimals to floats, how is that helping? I'm saying that if the underlying language was loosely typed, those kind of productitvy saps & bug fountains would not happen. In the few instance where you cared about type, a loosely typed language can usually offer something.
Last time I checked, enterprise ETL tools were sold as capable of a lot more than simple OLAP to OLTP. I find the reality provided is somewhat underwhelming. Given that facebook can tell the difference between a photo of Dave and one of Jim, why do I have to manually provide a mask for every single date field flowing through an enterprise?
Re:Airflow and the Future of Data Engineering: A Q&A
by busterarm on Wednesday March 01, 2017 @01:38PM
(#13766773)
I don't use enterprise ETL tools. I pretty much write my own every time and occasionally I'll supplement that with things like Kiba.
Re:Airflow and the Future of Data Engineering: A Q&A
by frugalmail on Wednesday March 01, 2017 @01:44PM
(#13766851)
We're part of a large enterprise but we use Azkaban and Scala for our data movement. And strong typing is a must in our group.