> corporations are just really bad at building incentives for long-term thinking.

Most corporations are really bad at long-term anything. CEOs want every quarter to be more profitable than the last, mostly everything else is irrelevant.

This reminds me of a post[0] where 3 junior devs had written a spaghetti monster that generated 20 million dollars annually, and OP was crying for a rewrite while everyone on HN chuckled.

"Long-term thinking"--often times, you ain't gonna need it. "Engineering" also rhymes with "bike shedding".

[0]: https://news.ycombinator.com/item?id=32883596

100% agree with the "building for trust". The only problem is we tend to hire based solely on technical skill. I don't recall an interviewer ever asking me something like, "how do you build a team that values trust and what does trust mean to you?"

Even when I joined the team, we never discussed trust and what we can do to be a more trustworthy team. It was always centered around how can we be more efficient in our deliverables.

This is the video I usually reference: https://www.youtube.com/watch?v=kJdXjtSnZTI

The pay for developers is also decidedly non-linear compared to ability levels. Promotions are famously non-deterministic.

Hordes of $300k developers trying to become $500k developers who are trying to become $1M developers.

"waterfall" is just a pejorative term for the kind of planning represented by a Gantt chart, which kinda looks like a waterfall.

This kind of planning is _necessary_ for traditional engineering projects where certain steps of the process can take months or years and requires scheduling of many resources.

Purely software projects usually don't need that because if your tools are good you can ship a completely new version in minutes to hours instead of months to years. You also get very easy do-overs if there are mistakes which you don't get if you're building a bridge or a factory.

That you can ship quickly is not really relevant though. Case in point is that you most often cannot change the architecture midways without permanent damage. That's why many software projects architecture looks like a building where the architect completely changed their mind after every floor, why everyone complains about technical debt and projects are often thrown away after a couuple of years and rewritten from scratch.

I'm not saying agile has no place, just that many projects would benefit from a little bit more foresight.

Also, I think successful agile projects are often those where the long term planning is there, even if it is not obvious or talked about a lot.

>I'm not saying agile has no place, just that many projects would benefit from a little bit more foresight.

There is really a spectrum of choices for how much you plan and tradeoffs with each level.

Success is finding the right balance for you, your team, and your product.

The phrasing I once heard (source sadly lost to my memory)

"If the physical properties of concrete changed every 6 months, then structural engineering practices would look very different."

But that's the world most software is developed and lives in.

"This avionics firmware is absolutely terrible."

"Well, last week it was driving a bus."

I'm reminded of a build break in Google web search because of a dependency on the self-driving car project. And that's how Bazel rules were given visibility lists.

Waterfall in software development was different than a Gantt chart based "project that was managed" like the construction of a bridge (an example close to my house). They added new requirements to this bridge (like they wanted the surrounding grounds to be maintained nicely, so added irrigation and gardening etc). But no new requirements were added to the foundational "here's a bridge and it needs to cross this stream from this point to that point" (high level).

Waterfall in software was continuously changing those fundamental parts. You'd be fine tuning the irrigation and needed to destruct the whole bridge because some new requirements came up. That's where the anti-Gantt and "bad waterfall" comes from, the reality is the software world as a whole moves way quicker and requirements are way more malleable, subjective than building a bridge.

>Waterfall in software was continuously changing those fundamental parts. You'd be fine tuning the irrigation and needed to destruct the whole bridge because some new requirements came up.

Exactly this same thing also happens in non-software "waterfall" projects as well. Mid-project fundamental requirements changes which result in having to re-engineer large parts of the project. This is one of the reasons military acquisitions are so expensive, there's a huge problem with requirements changes.

And thus us why large-scale construction companies can bid so low, because most of their revenue and profit come from the inevitable scope and requirements changes later in the projects.

Perhaps to some degree, but I've never seen a skyscraper that was almost done being torn down to be re-architected and rebuilt.

> This kind of planning is _necessary_ for traditional engineering projects

People keep repeating that, without any empirical evidence at all. If it were only software developers justifying a body of knowledge, it would be ok, but all kinds of people keep repeating that.

Yet, that kind of planning predictably fails every single time in every single engineering practice.

I can guarantee you, if an engineer designs the structure of a bridge, send it to a crew to construct, and go away to only test after it's done, the bridge will fall down before it's even built.

the fault is not allowing time to test and fix in the right places not in wanting a schedule.

The schedule also does never work anyway.

Passing a design down and going away doesn't work. Scheduling tasks before you star them doesn't work. Testing only at the end doesn't work.

The one thing civil engineering has that resembles PMBoK is designing things beforehand. And only civil engineering does that, only because they can't prototype.

The benefit of a Gantt chart isn't meeting deadlines, it is organizing work. You have all of the steps and what each of them depends on and a timeline. If something slips you know how to update your estimates for everything downstream. If several things are at risk you can see which ones have more critical time constraints and allocate resources effectively.

> If several things are at risk you can see which ones have more critical time constraints and allocate resources effectively

Hum... Have you used them that way? And have you needed a project design to actually discover risks and estimate that you need to act on them?

Or better, have you found any unexpected risk on a project design, and seen it materialize on practice?

> If something slips you know how to update your estimates for everything downstream.

And then your renewed estimates will be wrong again. No idea if more or less wrong than before your revision though. It can go either way.

>Hum... Have you used them that way? And have you needed a project design to actually discover risks and estimate that you need to act on them?

Yes. These things are all over the place in defense.

One of the most important exercises in doing a Gantt chart is finding the "critical path" which is the set of tasks which can't slip without effecting the final timeline. If step two has two separate tasks that step 3 depends on, the longer task is on the critical path, the shorter is not because the shorter task can be delayed at least some without affecting when step 3 starts.

As the project continues you update the chart with what actually happened vs. your plans. The goal is to minimize changes, sometimes you can, sometimes you can't, but when there are changes you can see how the whole flow is affected and you can easily communicate the change in deadlines, changes in budgets, you can pull people off of one task and put them on another, etc.

In agile you do way less of this planning and the path is usually much more simple so you don't _need_ these tools to be successful.

civial engineerin, is often small variation on every other structrure. as suchiit is easy to estimate as things didn't change. when things are very different is when they overrun schedules.

> "you can ship a completely new version in minutes to hours"

Not all that long ago, software was distributed on physical media such as floppy discs and CD-ROMS.

In my startup, for all the future devspeed promised by developers who pretend to know how to plan, after 4 years we sure lack to see their features.

I don’t know, it feels like customers buy our product because has features, but who doesn’t like a good K8s stack.

The market never seems to end producing those developers who spend their life refactoring and never shipping anything, nor taking responsibility for the fact that their pile of technologies SLOWS DOWN the other developers while never solving the problems they were implemented for.

It’s not a manager problem. Developers don’t like taking responsibility for their mistakes.

Related: I don't know why software estimation doesn't spend more time on risk and less time on estimation.

How much risk (to delivery) is left out there? What can we do to reduce that risk?

It's a known thing, but seems rarely used in software shops. https://en.m.wikipedia.org/wiki/Spiral_model

Who cares if 97% of the code is complete, but the remaining 3% is a critical feature, with unclear requirements, that's never been prototyped, and has no test data?

The challenge is the mindset in most software shops of "We're not building a space ship". Because of the ability to rapidly iterate, there's a sense of invincibility. As we've seen, however, some aspects of the product should in fact be treated like building a space ship. For instance, with something like properly managing data (so you don't get breached), a detailed risk assessment and robust strategy should be more highly prioritized. For displaying a UI to end users, perhaps not so much, but even then there are accessibility concerns that a good product would address.

Safety/security risk is important too, but I was suggesting development keep an eye on delivery risk.

I.e. the risk that a project won't be delivered with the expected features and by the expected end date

Fundamentally, time estimates are useless without a coupled risk/certainty metric.

If you ask me how long something is going to take, and I believe it'll take somewhere between 3 months (worst case) and 2 weeks (best case), what number do I reply to you with? And how do you interpret that number?

And how does my reporting change as we get closer to delivery?

Reducing the size of work being estimated band-aids the problem, by shrinking the absolute magnitude of the bounds, but doesn't fundamentally solve it.

Common heuristics like "double how long you think it's going to take" are other ways this appears.

I think we’re in agreement. In my mind, you can even get delivery risk from a safety/security standpoint. As in, the product should be developed on time while accounting for the security/safety stuff. And you get there by placing risk front and center for the entire system from the beginning.

> It’s not a manager problem. Developers don’t like taking responsibility for their mistakes.

I’m glad you agree on both of these points. Like I said, it’s a problem with building the right incentives, and a problem with people who are trying to avoid blame.

A developer with a good, long-term view has a light touch. On one side, you have reckless developers who ship out buggy products that cause problems for operations and customer trust. On the other side, you have fastidious developers who just keep refactoring everything and never ship features.

Sometimes, you find a team that can do both. When I’ve seen this, the team has been made of both types of developers. A mix of skill sets, and a high level of trust that they won’t be blamed for failures (outages, missed deadlines). Your developers who move fast and break things will take the time to write a few tests and break things less. Your fastidious developers will relax their standards a little bit and push out code faster, knowing that their contributions are still valued.

Of course prioritizing shipping more features is going to ship more features. The whole point of prioritizing engineering is to trade off some speed for other things, like your shit not breaking. The incentives in most companies produce catastrophic failures because the mechanisms for punishing individual failure on speed metrics (the easiest metrics and thus the only metrics most organizations know how to or care to measure) are powerful and numerous but the organization-level incentives don't adequately punish the kind of catastrophic failure we are seeing constantly. That cost is externalized from the organization that failed. Their customers and those customers' customers and anyone affected by the whole world being made of brittle nonsense pay the price and have meager recourse if any. If your reason for prioritizing correctness over speed is because you believe that this will eventually produce more speed, I don't know what to tell you. That's possible and a really nice outcome when it happens, but it's not the point of doing it and it's not the typical benefit

Here’s how I’ve seen this play out at some larger, more bureaucratic companies. (This is just an example, it doesn’t have to play out this way.)

You start by moving fast and putting out features. Then something breaks in a high-profile way and the hammer comes down. The environment changes and people start avoiding blame. Some of the effects I’ve seen:

- You bring more people into meetings (diffuses the blame when something goes wrong, but slows everything down)

- You demand more detailed requirements (makes development more predictable, but good designs tend to evolve and do a lot of discovery)

- Tedious work doesn’t get automated, and your headcount requirements go up (because tedious work is predictable, and automation is a risky project)

Eventually, somebody in senior leadership (hey, I said “larger company”) notices that the entire org has basically slowed to a crawl and is constantly demanding more headcount, and they have to go in and fix things. Or they decide to axe a bunch of projects, or reorg, or something else.

I want to emphasize that this is a possible way that the story plays out, not that the story always plays out this way.

> Of course prioritizing shipping more features is going to ship more features

I mean, I'm not entirely sure on that. Prioritizing shipping features may make a codebase be impossible to understand, so that any additional features take longer and longer to implement. Prioritizing design may make a codebase be easier to understand, such that each new feature can be added with breaking existing functionality.

There's a quote I like from Jonathan Livingston Seagull: "It's strange. The gulls who scorn perfection for the sake of travel go nowhere, slowly. Those who put aside travel for the sake of perfection go anywhere, instantly."

>The market never seems to end producing those developers who spend their life refactoring and never shipping anything, nor taking responsibility for the fact that their pile of technologies SLOWS DOWN the other developers while never solving the problems they were implemented for.

The market absolutely produces those developers, it's just that their salaries start at 7 figures and go up from there.

> I don’t think this is anywhere close to an accurate account of history.

This was my thought also. I've been building software systems since the early 80's and I've never been part of a project that looked like waterfall, but I've also been in small to midsize companies, not large.

The approach on projects has been somewhat consistent for my entire career which is just a pragmatic look at the project and approach the problem based on the nature of the specific project. There are some projects or parts of projects that really require up front design and planning (e.g. when coordinating changes in multiple external systems that must all play nicely together) and some other parts of projects where the best approach is to iterate (e.g. we've never done something like this before and can't use our experience to guide us very much so we need to try some stuff). Projects are typically a combination of all of the above and more.

When I originally read the agile manifesto my thought was "ugh, ya, this is all pretty normal common sense, sounds like these guys must be working in some large corporations where the bureaucracy has taken over and they are trying to push back."