You obviously can have modularity with shell scripts:

run_all_etl.sh

build_internal_dashboard_etl.sh (called from run_all_etl.sh)

build_external_dashboard_etl.sh (called from run_all_etl.sh)

How many models are in "run all etl", and what if one fails? You can't see how issues can arise here, with what you describe? I mean, go ahead and write your SQL pipeline in bash. Many of us are using dbt because we see a lot of benefit, we aren't all morons.

> How many models are in "run all etl", and what if one fails?

hundred tables. one fail -> whole pipeline fails to ensure e2e and overall consistency.

No you misunderstand. If you have some kind of dependency tree that's like:

    A        B        C        D
        1        2        3
        i        ii
    Ai     i-ii

...and you want to only rebuild Ai and anything that it might rely on, all you do is `dbt build +Ai`. If you want to rebuild B and update anything that relies on it, all you do is `dbt build B+`

> You obviously can have modularity with shell scripts

Just as kind of a meta point, I think you're assuming that people are introducing a lot of incidental complexity into this problem, but I don't think you're open to the possibility that there's a lot of essential complexity here. I'm talking about DAG tools and the features that hang off of them, and you're saying "hey there's new thing called shell scripting". That's not useful in this conversation. Maybe going forward, let's assume people know about Bash.

In my example ETL is divided into modules with clear scope and dependencies, as I believe should happened in software practices.

If you have kitchen sink of 1000 tables chaotically connected to each other, then yes, you need DAG management tool to manage chaos..

> If you have kitchen sink of 1000 tables chaotically connected to each other, then yes, you need DAG management tool to manage chaos.

This describes 99% of data teams, so I think we agree!

> This describes 99% of data teams, so I think we agree!

there is a chance that dbt incentivies data teams to go that route, otherwise they would think about modularity and ownership more.

I'm a Vim user and I've made this point when I rail against IDEs; it's even on the front page today [0]: "5. The belief that complex systems require armies of designers and programmers is wrong. A system that is not understood in its entirety, or at least to significant degree of detail by a single individual, should probably not be built."

I think I agree, but I think there are a lot of things you could say about this. What if you're just starting out and you don't know how to efficiently design systems? What if you work in an industry of people under 40 (under 30?) who by definition can't have the kind of experience that would let you design small systems? What if you work for a company that doesn't want to give you the time to figure it out? What if you have a competitor that's just shipping trash and forcing you out of the market with a "we'll clean up the tech debt later" mentality? How do we make tools that are powerful enough to enable us to do hard things, but wise enough to keep us from doing foolish things? Should we rely on our tools to do this? Is this an argument against centralization? Isn't that deeply inefficient?

Honestly I struggle with this all the time. I've got the outlines of some ideas and there are some tech communities grappling with the same issues, but I think we're just at the very beginning of a rethink about how we do software.

[0]: https://liam-on-linux.dreamwidth.org/88032.html

> How do we make tools that are powerful enough to enable us to do hard things, but wise enough to keep us from doing foolish things? Should we rely on our tools to do this?

IMO: No we shouldn't. And that's why I'm in the "Think about it yourself, bulid your ETL with a clean structure so you know what your 1,000 [SQL|shell] scripts do" in stead of the "Let your tool figure it out for you" one.

> Is this an argument against centralization? Isn't that deeply inefficient?

Or for centralization, if we hark back to the "You can get a loooong way on a single server" argument above. Depends on how you define "centralization", I suppose. (Seems to me everything always depends on your definitions, in the end.)

But wait, does this still have anything to do with IDEs? As opposed to Vim (or, say, Notepad)? If so, I'd say IDEs also belong to the category good centralization. At least older ones, before they became desktoppified Web apps with baked-in AS (Artificial Stupidity)...

> Honestly I struggle with this all the time. I've got the outlines of some ideas and there are some tech communities grappling with the same issues, but I think we're just at the very beginning of a rethink about how we do software.

On that debate I'm not so much conservative as... Reactionary.