News for hackers. Stuff that matters.

A comment on What Is Dbt and Why Are Companies Using It?. Parent comment

Re:What Is Dbt and Why Are Companies Using It? by jeanlaf on Thursday December 02, 2021 @10:22PM (#29426294)
Might be worth looking into Airbyte!
[ Parent | on HN ]
Re:What Is Dbt and Why Are Companies Using It? by thejosh on Thursday December 02, 2021 @10:48PM (#29426422)
Airbyte looks great, and the UI is fantastic.

It uses Java for its connectors, and looks great but has issues importing a massive dataset into S3 as there is a chunk limit of 10k, and each chunk size is 5mb :).

[ Parent | on HN ]
Re:What Is Dbt and Why Are Companies Using It? by jgraettinger1 on Thursday December 02, 2021 @11:44PM (#29426700)
We've used https://github.com/estuary/connectors/pkgs/container/source-... to load data sets in the many terabytes. Caveat that, while it's implemented to Airbyte's spec, we've only used it with Flow.
[ Parent | on HN ]
Re:What Is Dbt and Why Are Companies Using It? by jeanlaf on Thursday December 02, 2021 @11:55PM (#29426764)
That’s the current focus of the team. Consolidating those connectors :)
[ Parent | on HN ]