Re:Bauplan – Git-for-data pipelines on object storage
by tech_ken on Thursday April 17, 2025 @11:25AM
(#43718254)
Wow that's super cool, thanks for explaining! I will definitely keep an eye out; having that level of certifiability and replayability at a pipeline level is something my team has been having too much angst about. It would be incredible to be able to abstract it away as cleanly as we do code management.
Re:Bauplan – Git-for-data pipelines on object storage
by jtagliabuetooso on Thursday April 17, 2025 @01:47PM
(#43720066)
Awesome, you can write me anytime to geek out ([email protected]) or follow us for more community sharing (papers, deep tech blog posts: https://www.linkedin.com/in/jacopotagliabue/).
The full auditability is already here today though, so we are always happy to hear your feedback on our public sandbox (you can join for free from our website, and reach out to us at anytime for good and bad feedback!)
You can automate many changes / tests by materializing the parent(s) of the target table, and use the SDK to produce variations of a pipeline programmatically. If your pipeline has a free parameter (say top-k=5 for some algos), you could just write a Python for loop, and do something like:
client.create_branch() client.run()
for each variation, materializing k versions at the end that you can inspect (client.query("SELECT MAX ...")
The broader concept is that every operation in the lake is immutably stored with an ID, so every run can be replicated with the exact same data sources and the exact same code (even if not committed to GHub), which also means you can run the same code varying the data source, or run a different code on the same data: all zero-copy, all in production.
As for the semantics of merge and other conflicts, we will be publishing by end of summer some new research: look out for a new blog post and paper if you like this space!