That sort of exploratory work is really good because you’ll end up reading relevant documentation, then actually trying to use whatever it is you’re learning about. Often I’ll throw together a hacky version of what we’re aiming for, but ignore anything but the happy path. No error handling, no UI beyond the bare minimum, just the very core of what’s being done.

Easier said than done though

Or, I could deploy something to QA and ask for feedback and forgiveness. Then a 3 week discussion becomes a 30 minute demo + a few small notes.

https://www.youtube.com/watch?v=PUWg905fGTA

Both funny and insightful for anyone starting a project.

If so, why do we want to dive so much into requirements that are likely to change anyways?

Not saying we should jump into design without understanding the requirements, but sometimes requirements are incomplete or not clearly defined and we cannot sit and wait until a new revision is out or someone becomes available to answer our questions.

I guess this really depends on the area we operate.

In my business domain we call requirements "wish lists" as the solutions we build are for complex hardware that generally doesn't exist yet and there's a good chance the requirements spec you got today will be rewritten in six months, yet we need to start working as soon as the project is launched.

So we have learned to design things that are reasonably easy to adapt, replace and test, we even developed an entire framework for this very purpose that has become some sort of spinoff business as some customers wanted to use it for their in house development.

Who says all the requirements are likely to change anyway? That’s a whopping great hypothetical on which the entire premise rests, but YAGNI applies to development processes as much as to code!

Yes, requirements often evolve and we often start with some gaps or ambiguities in the spec. However, that’s a very different situation to having no detailed requirements at all. In one case, you know where you’re trying to go right now and can set a course that will take you there. Even if the destination is changed a bit during the voyage, you can adjust your course to compensate and still be heading in roughly the right direction. In the other case, you literally don’t know which way to start.

Moreover, jokes about skyscrapers turning into spaceships aside, the central requirements for a software project usually evolve relatively slowly. Business priorities or some of the details might change faster, but in most fields the underlying domain concepts and actors that the software models are going to be relatively stable.

Big pivots where you’re effectively trying to build a completely different product afterwards do happen. However, a team that pivots so often that it never really understands where it’s trying to get to before changing direction again won’t last long because it can never get far enough in any direction to deliver real value.

It takes skill and effort to abstract from these detailed wishlists into generalised primitives which remain somewhat constant. This work is similar to the fuzzy logic social scientists work with, and that's exactly the sort of thing most techies are not very good at.

100% agreed. It’s well worth doing it if you have the chance, though. Understanding how your customer sees their world and what problems they’re really trying to solve is valuable both for building a useful product and for keeping that customer happy!

It's always a fuzzy nebula and the only way to get insight is to build some prototype, see how it behaves and improve upon it. With the inevitable "suboptimal" early design somehow enduring and adapting through successive iterations, never have time or be able to afford a full rewrite from a business perspective: stuff works for the client, regardless how terrible the engineers view the build process. So people love to lament how awful the early choices were and how far in outer space we'd be, should some shiny late-stage idea have been applied. But it's a lot more like the x86 CPU architecture, in spite of all the detractors, it endures.

Ahh, and I've also worked for a company that did rewrites. A lot of them. Like every time the product reached some 80-90% of functionality, they'd throw everything away and redo from start. Never shipped anything in my entire stay there. Again in my life I never seen such a thing, it's like the managers were not right in the head and developers were lacking any common sense. So it shouldn't come as a surprise that now (after some 5+ years of this) they closed and fired everyone.

Money and time (for us humans) are finite, big things at lease have to be targeted precise efforts, a good plan and architecture that maps back and supports the requirements.

Which is precisely what agile is about, specifically the estimation process. No story enters the sprint unless there is unanimous consensus among the engineers as to its point value, and it's the process of coming to that unanimity that makes sure each individual understands what needs to be done.

A lot of people like to complain that agile slows down their coding, and this is by design -- to achieve exactly what you're asking for.

> But now we often seem to have one-paragraph tickets, maybe spend a few minutes thinking of some class names that look about right for each one, and then dive into coding.

That isn't agile. If you're estimating a new story the team hasn't discussed before, it's usually going to take between 5 and 30 minutes to come to consensus on its size and what it is conceptually.

So the main problem you're describing is what Agile solves.

Then you seem to have a secondary problem which is about different team members working in incompatible ways in terms of coding architecture. That require a team lead who is in charge of architecture decisions, that team members consult with whenever they start a story that there's no precedent on how to build. And code review ensures code is written as planned.

I call bs.

Are you telling me that the less experienced / less confident devs on the team don't cave to pressure from the team lead or whoever happens to be the most forceful / loudest to agree that, ok, this is only a 5 pointer and we all understand what needs to be done, because DAVE says it's easy, 30 minutes work tops, and DAVE is tired and has better places to be.

You write as if the team is all top engineers with equality and not one leader and his minions.

There's nothing difficult or impossible about it at all.

But it's true that if the leader doesn't want to follow it that it's not going to work. It requires the person in the position of authority in the room to buy into it.

Software is complicated. Half the time I don’t know what the hell anyone is talking about due to labyrinthine requirements, but I’m not going to stop the meetings to ask questions unless I’m the main guy working on it.

All of the points you're raising are solved inherently by the way points estimation works with the points playing cards.

You're not stopping the meeting. But it will be your turn to explain why you think it's the number of points you chose, so you'll do that.

It sounds like you have a very dysfunctional team if you're characterizing it as a team leader and his "minions".

I'm sorry that's your situation.

Some larger corporations have agile facilitators precisely to nip this kind of problem in the bud.

Do you know what planning poker is, and are you aware it's not a game for fun?

I have heard some third tier companies and lower play "planning poker" at work. It doesn't sound like any fun or efficient but a cheap paycheck for an "agile helper". I am not sure how many of them have remained employed during these layoffs.

Are you against those things? "First tier" companies use planning poker as well, you know.

You seem to be oddly hostile to the concept without seeming to understand it at all.

There's nothing to understand, the agile manifesto or worse the "scrum" pamphlet can be read on a lunch break. Somehow engineers who put ten-thousand hours into their craft need a layman agile helper to understand it?

Worse, I never seem to get those abstractions quite right until I’ve gotten them wrong a few times. If you don’t take the time to iterate and build some knowledge of the solution space, anything you make will be a bit of a buggy, fragile mess.

And I think it’s much easier (and more efficient) to do that iteration during development than spend years building something messy and then try and remake it from scratch, cleaner, later. But that means it takes more time to get to market, and that’s a really hard pill to swallow (for good reason).

Usually the individuals doing the work do understand that, but it is really hard to properly divide work and describe requirements. Why people try do divide it in even smaller tasks which increases this kind of complicated creative-writing work is beyond me.

> We realised that waterfalls that start with months of requirements capture followed by months of BDUF aren’t a great idea.

What we should have realized by now is that we never did waterfall right because it's inventor Winston Royce required it to be done TWICE for each project. Two iterations.

What this world needs is Atlassian hiring a number of work psychologists and theorists and have them streamline JIRA based on their learnings. Not in a UX "I want to create a story"-story sort of way. But in a "How do I deal with 50 tickets that are vastly different in quality?" sort of way.

They should answer questions like: "How can we make it so that the user always immediately knows whether to put this text in JIRA or confluence"?

Then Microsoft needs to hire the same people and improve text creation in outlook.

Reading the original waterfall paper was eye opening for me. I saw that we're falling into patterns that are decades old. Someone insightful tries to explain that learning and building are two sides of the same coin, and companies read it as "ok, I need a learning team and a doing team" and run with it in the wrong direction. Taking the conceptual structure of some paper of manifest and applying it cookie-cutter style is missing the point entirely.

What Conway and Royce were trying to convey is that the social process that underlies software matters. What people do with it on the field is using the examples employed in the explanation and mistaking them for the solution to their lack of respect for the intricacies of social processes.

(Yet another reason why LLMs are so limited: they take their earlier writing as sacred and unchanging as a direct consequence of their autoregressive structure.)

In my decades of experience in multiple companies doing 'agile', that's something that we've always ostensibly strived for, but in practice it almost never happens in reality. At least not as intended. Occasionally it happens, but only by pure chance.

It seems the 'agile way' is to attempt to manage a project by taking a sizable but coherent and cohesive concept/feature/requirement/design/plan, shatter it into a thousand incohesive random fragments of varying sizes (all estimated to be bite-sized, but true size unknown until they're actually done).

Then everybody grabs a few fragments and works on them, placing them somewhere more-or-less where they're hopefully supposed to be, hoping that in the end everything will come together somewhat similar to the picture on the box. Then everybody has to try to duct-tape and glue them all back together again on a herky-jerky schedule, like doing a puzzle with constant interruptions and distractions for the agile rituals.

The results aren't necessarily worse than waterfall/BDUF, but not necessarily better either. There is in truth a lot more potential to course-correct along the way. But since everyone's spastically going different distances in different directions at different speeds on as-yet-unconnected fragments of varying sizes, and no one can see the big picture, that theoretical course correction en-route is not nearly as powerful as it sounds.

I've seen some debates about shorter vs. longer sprints. The die-hard agile people tend to push for the shortest sprints possible or even shorter, leaving no time whatsoever to clarify requirements, work out integrations, or do any QA/testing. No time for thought at all, just spit out some code, any code, as fast as possible and then it's review/demo/retro time. The veterans push for longer sprints, with time to understand and coordinate, experiment and test. But that gets a lot of pushback for not being agile enough, so we usually end up somewhere in the middle. And a lot of things don't fit neatly into that timeframe.

Which in my opinion is not the worst thing, but also nowhere near as efficient or as effective as it could be if things were not broken into bite-sized chunks with arbitrary deadlines, but instead developed as parts of a cohesive whole. Whether from the foundation up (risky) or by building vertical segments and linking them together.

"Move fast and break things" indeed. That's the state of the art - breaking things as fast as possible. We could do better.

I think it should just be accepted that it is not possible to know. Unlike building a building, you cannot know. It is possible to do a feasibility study, or a prototype, and time box it, but that is as good as it gets.

The stakeholders don't want to hear this unfortuantely. They refuse to believe it isn't possible to know ahead of time how much work is needed. In order to appease, the engineers and managers have developed a set of "practises" such as agile, but in the end, it is a lie.

For the rest of us “slow is smooth” means you should work on something that is analogous to the most important functionality first, to build your ability to do that work. Then as you gain skill, tackle harder and harder problems.

Sometimes the rule of three lets you go back and fix so,e of that early poor ordering, but that doesn’t always trigger at all and sometimes triggers when you still don’t know what you’re doing. And then everything you do later is hampered by getting things wrong up front.

Some times those are the same, but often the part of a project that sinks it isn't the core functionality but some prerequisite that turns out to be much harder than anticipated.

In an 18 month or three year project, it doesn’t affect your deadline much if you do the hardest bits in month one or month four. Except in how much rework you end up doing to those parts that have already started to cement bad habits.

When we have a Rule of Three situation in the product backlog, I usually recommend one of two strategies. Do the hardest one second, or do it third and allocate very little time for the first two - you’re going to rework the whole thing when you get to #3 anyway.

On one project, no matter what order we did them in, the third one took the longest time. Even when we tried to take our time on the second. Took me a year to train them to stop doing that. I’ve never seen a team so sure of their abilities yet with such a low opinion of their product. Big ol’ echo chamber.

Another phrase with similar meaning (and interesting history) is "festina lente" ("make haste slowly," in Latin). I named my company after it (:

It's certainly an ancient concept. I feel like software is ultimately about humans organizing complex activities, and humans haven't changed that much in the past few millennia, even if the activities have gotten more complex and abstract.

I feel like the ones-size-fits-all approach of trying to say approach X or Y is superior (eg: scrum vs. kanban vs. waterfall, etc) is totally off the mark. Different sorts of work require different sorts of effort. Creative and full-of-unknowns work is different from assembly-line sort of work is different from maintenance work, etc.

On a meta level, I think part of the "slow is smooth..." philosophy is to take a deep breath at the beginning and think about what sorts of approaches are going to be best suited for the tasks at hand. As opposed to some mindless "scrum is in, so we'll do that for everything". Or "everyone is using NoSQL now, so we should too" or any number of other cargo-cult-y behaviors.

Festina Lente!

Organizations are much like software: embodiment of domain knowledge. It needs to be codified, and if you're not actively cultivating it locally, you're outsourcing the one thing that matters: learning.

It's a 100% in the interest of incumbent players to divulge ready-made pretenses of universal solutions. It precludes competition.

> Be first, but first be right.

"Agile" coaches scammed you

When I am under time pressure I slow down a notch. Over the years my slow speed has gotten faster as well.

I've been in enjoying the converse of this i.e. "fast is smooth". It was provoked by using an ultra-fine fountain pen nib - too slow and it would be scratchy and dig into the paper with poor ink flow but using it very fast and lightly and it would be really smooth and delightful.

I am prone to perfectionism, overthinking, procrastination, majoring in the minors and getting derailed by little hiccups. I get motivation by seeing progress. If I pause I can struggle to get started again so momentum is really important or get into a negative spiral.

For my personal productivity, it helps to force a pace and do things faster than I naturally want. I need to keep up momentum and quickly do the known things and discover the problems that need to percolate in the back of my mind to reach a creative solution.

I guess we all need productivity epithets that are counter to our dysfunctional traits - too slow then speed up! too fast then slow down!

The waterfall:bad, agile:good trope is really tiring. Like listening to a teenager that knows jack shit explain how the world works.

This usually takes the form of pattern driven developers: the developers that must fit the entire problem into a predetermined pattern (which is ridiculous because every problem is unique and there are very few patterns that are truly applicable everywhere). This will also take the form of Amazon's famous 6 page design upfront. It's funny, they wanted me to prescribe what the expected latency of a system that would be running over an unreliable network was and figure out an SLA based on that, before I wrote an ounce of code! How in the world are you even supposed to begin getting accurate measurements, without measuring!

So, planning is good yes. But the reality is every problem is unique. Unless you have somebody who's solved the exact problem you're trying to solve, planning is going to do no good and you're better off exploring first. Once you became a domain expert in your tiny niche domain you just explored, then you get to make a plan.

But enough with this nonsense that we can magically figure out what the exact requirements are for a problem before we've even begun to measure the appropriate metrics associated with the problem. And one of the key ways we can measure stuff is by exploring the problem space with code.

The bonus is, being a Python notebook, you usually avoid the common pitfall of shipping crappy prototype code to production; you are forced to rewrite it because putting a Python notebook into prod would make any software engineer shudder.