hugocorreia90

First time I heard about Rocky Linux distribution. Thank you for the heads up.

Yes, I can see you're a bit late indeed. Several replies ago I have already acknowledged that I used LLM for the initial post and initial replies.

I use AI everyday, for different purposes, fixing my English when I feel I need, brainstorming, automate tasks, getting ideas for dinner, asking things I don't know about raising a 1 year old boy, eck even finance stuff. And to be honest, I'm quite pleased about it and see no shame on it. I'm no salesperson, nor I have a grasp at marketing, nor I'm used to promote my work. With this thread in HN, (which was a suggestion by LLM! :D), I just wanted to share what I built a let others use it, regardless of typing each character or not, or using LLM, or using smoke signals.

But one thing I cannot avoid, is being polite and friendly because that is who I am when speaking in my native language or in English with my peers. So, saying "Hey" and "Hi", "I appreciate" and "thank you", is part of my day to day.

I didn't know about the article you mentioned but thanks for letting me know. :)

I hadn't come across Bauplan or your work before today's thread. Looks like a few of us are landing on branches/replay/lineage from different angles (like yours Iceberg-native, mine warehouse-delegated). I will spend proper time with the paper and Bauplan.

Hey jtagliabuetooso! Rocky is SQL-first instead of Python-first, and that was a deliberate scope choice. Also, Rocky acts as control plane that delegates compute to the warehouse (Databricks, etc) rather than owning the runtime. I don't have a strong use-case for nested branches, so no, that's not a feature Rocky has of today. I'll read the paper, thank you for sharing.

Hey jtagliabuetooso! Absolutely, the idea isn't new. Rocky's bet is on the shipped implementation. I'll read your paper, thank you for sharing.

Love your pet names and how you characterize them if you would name something after them :) Wiley is an interesting name!

I have another side-project, still private, which I named Shimi, my current dog's name. I'd thought naming my dog Sashimi, but Shimi is just shorter and simpler. I'm now considering stealing the name from my this other side-project for renaming Rocky, but I'll put more thought into it :)

Make sense. Reviewing the README is on my TODO list. Thanks for the heads up!

Oh yeah, good call.

Accepted! :)

Hey Xiaoher-C! Hum, I don't have a lineage diff command yet. As of now, I can make a small lift to wire up together two commands that already exist: "rocky ci --diff --base main" which runs a diff between main and HEAD, and "rocky lineage --column columnname --downstream". I'll add this to my backlog!

Thank you!

I truly appreciate your feedback and it's definitely a lesson learned for me. As I said to cmrdporcupine, "The only reason for passing my replies through AI was just because it's my first time posting here and opening a side-project of mine publicly. "

All the engine architecture decisions are mine though and this project came up to solve a real problem in a data pipeline that serves multiple clients, connectors, producers, etc.

Hey Anders! Thanks a lot for dropping a comment and show interest in Rocky. Yes! I won't going to lie that Jinja is one of the things that gives me some itches :). But it wasn't the major reason for start building Rocky though.

It all started with the need for auto-generating dbt models from the FiveTran connections I integrate with, then having to hot reload code location in Dagster to discover new assets. All in a zero-touch data pipeline. FiveTran connections are discovered as they're created, assets are materialized as these connections sync.

Auto-generating these dbt models and get the manifest aligned between Dagster code location reloads plus spinning up pods in EKS for each Dagster runs that need to rely on these auto-generated models have some impact on the performance overall, not only in production, but also affects DX in their local environment.

Rocky wasn't born with a "dbt replacement" in mind at all, but it was born to solve a real issue I'm facing. I made sure I can integrate well with dbt as it's in my plans to leverage the awesome work available as dbt packages for FiveTran.

I'll definitely have a look the crates you mentioned! Thank you!

I totally acknowledge that. The only reason for passing my replies through AI was just because it's my first time posting here and opening a side-project of mine publicly.

All the engine architecture decisions are mine though and this project came up to solve a real problem I currently have at work with a zero-touch data pipeline leveraging FiveTran, Dagster, dbt and Databricks. This is a data pipeline that servers multiple agencies and data producers who work with data from more than 300 clients and multiple connectors.

Rocky essentially was built based on all the time spent awaken at night thinking about all these problems and how could they be addressed differently, considering that dbt is not suiting well this particular use-case.

I decided to open Rocky to public for free because of two simple reasons: 1st is that it might help others and I fullfill my ego of having built something other people like and use. 2nd is that I'm the solo maintainer. A project can only get proper traction if more people contributes to it.

Thanks Peter. All my side-projects are named after my pets. I had a dog named Rocky and given this project is also an underdog competing with well-established tools such as dbt and sqlmesh, I decided to keep Rocky when opening it to public. But I'm happy to get some suggestions for a better name to this tool :)

Fair. I just use it for tidying up my replies as I'm not a native English speaker.

Thanks for the careful read. The "what breaks if I rename this column" question is exactly what column lineage from the compiler is meant to answer, and you said it better than I did in the post.

On the schema-grounded AI angle: agreed. The failure mode you describe — structurally valid SQL that joins on the wrong key or aggregates at the wrong grain because the model hallucinated a relationship — is exactly what the compiler is positioned to catch. AI-generated SQL runs through the type checker before it can land, so suggestions that don't validate against the actual DAG never reach the user. NL-to-SQL tools that integrate a compile step would close exactly the gap you're pointing at.

On your two questions:

1. Branch isolation for stateful models — mixed answer, and worth being honest about:

   - Incremental: isolated. The watermark `state_key` includes the resolved schema, and `rocky branch create` swaps the schema prefix. So a branch run reads/writes a different redb key than main and they don't advance each other.

   - Snapshot: not yet. Today `rocky branch create` only writes a branch record; it doesn't copy warehouse tables. A snapshot model on a branch starts with an empty table (CREATE TABLE IF NOT EXISTS in the branch schema) and accumulates from the first branch run, with no inherited history from main. That's the gap. The fix is the next wave: native Delta SHALLOW CLONE / Snowflake zero-copy at branch creation, which gives point-in-time snapshot semantics without copy-on-write overhead.

2. Cost attribution. Both bytes scanned and duration are captured per-model in the run record (`bytes_scanned` and duration on `RunRecord`). Budget gating today is on cost (USD) and duration — `max_usd` and `max_duration_ms` in `[budget]` blocks in `rocky.toml`, as independent thresholds. A direct bytes-scanned budget threshold isn't gateable today; the bytes are in the run record for analysis but you can't currently fail CI on "this run scanned more than N TB". Reasonable extension if there's demand.

   To your Snowflake point: the warehouse-size × duration credit model and the scan volume tell genuinely different stories, so they're tracked separately rather than rolled into a single number.

Thanks. On model versioning — what's the use case you have in mind? A few options that map to different designs:

- dbt-style semantic-layer versions (v1/v2 of a model) - schema migration history - branch-based (Rocky already has branches + replay)

Different design choice for each, so it helps to know which problem you're trying to solve.

ClickHouse is tractable through the Adapter SDK without engine patching. If you can share roughly your model count and workload shape, I can put a real timeline on it. Open to community PRs too.

Fair point. Databricks owns a scheduling DAG (Workflows, DLT). What I meant by "owns the DAG" is the semantic DAG: model-to-model dependencies with column-level types that the compiler builds.

Workflows knows task A runs before task B. Rocky knows `dim_customer.email` flows from `raw_users.email_address` through three CTEs in `stg_customers`. Different layer, same word.

I'll be more careful with that framing.

Yeah, fair pushback, and yes the intro was AI-assisted. Marketing is not my strength nor I am a native english speaker. I built this in about a month with heavy LLM tooling and the seed comment is part of that. I'm not going to pretend otherwise.

The code is what it is. `cargo test --workspace` runs across 19 crates. CI on 5 platforms (macOS ARM/Intel, Linux x86/ARM, Windows). JSON output schemas are codegen-checked in CI so docs can't drift from the binary.

If you want to skip the marketing copy and look at engine reasoning instead: PR #240 (audit trail), #241 (column classification + masking), #270 (failed-source surfacing in discover).

I'd rather hear "the code is bad" than "the post sounds AI-written".