News for hackers. Stuff that matters.

A comment on Building an Open, Multi-Engine Data Lakehouse with S3 and Python. Parent comment

Re:Building an Open, Multi-Engine Data Lakehouse with S3 and... by bradhe on Tuesday February 18, 2025 @05:07PM (#43095749)
I'm one of the cofounders at Tower, posted this because I thought some people would be interested in the topic. Would be interested to know what Airflow is really...doing...for you here? Is it just an execution engine for your sqlmesh? Anyway, as we're trying to build out Tower would love to know more.
[ Parent | on HN ]
Re:Building an Open, Multi-Engine Data Lakehouse with S3 and... by dogman123 on Tuesday February 18, 2025 @05:30PM (#43095938)
sqlmesh execution engine + cloud resource provisioning

provision spark on emr or duckdb on beefy ec2 -> run sqlmesh -> wipe resources.

i'm still in MVP phase of revamping my company's current data platform, so maybe there are better alternatives -- which i'd love to hear about.

[ Parent | on HN ]
Re:Building an Open, Multi-Engine Data Lakehouse with S3 and... by bradhe on Tuesday February 18, 2025 @05:37PM (#43096012)
Cool, would be happy to talk a bit more about it. Want to shoot me an email (in bio) or if you sign up for our waitlist on https://tower.dev I can reach out in the AM (we’re in Germany)!
[ Parent | on HN ]