Bauplan – Git-for-data pipelines on object storage

Re:Bauplan – Git-for-data pipelines on object storage by barabbababoon on Wednesday April 16, 2025 @12:31PM (#43707384)

Plenty of stuff in common with dbt's philosophy. One big thing though, dbt does not run your compute or manage your lake. It orchestrate your code and pushes it down to a runtime (e.g. 90% of the time Snowflake).

This IS a runtime.

You import bauplan, write your functions and run them in straight into the cloud - you don't need anything more. When you want to make a pipeline you chain the functions together, and the system manages the dependencies, the containerization, the runtime, and gives you a git-like abstractions over runs, tables and pipelines.

[ Parent | on HN ]

Re:Bauplan – Git-for-data pipelines on object storage by korijn on Wednesday April 16, 2025 @06:30PM (#43711057)

I see, this is a great answer. So you don't need any platform or spark or anything. Just storage and compute?

[ Parent | on HN ]

Re:Bauplan – Git-for-data pipelines on object storage by jtagliabuetooso on Wednesday April 16, 2025 @07:30PM (#43711443)

You technically just need storage (files in a bucket you own and control forever).

We bring you the compute as ephemeral functions, vertically integrated with your S3: table management, containerization, read / write optimizations, permissions etc. is all done by the platform, plus obvious (at least to us ;-)) stuff like preventing you to run a DAG that is syntactically incorrect etc.

Since we manage your code (compute) and data (lake state through git for data), we can also provide full auditing with one liners: e.g. "which specific run change this specific table on this data branch? -> bauplan commit ..."

[ Parent | on HN ]