S.M.A.R.T, right?

Wrong!

Goodhardt's law. Once you put any sort of pressure on the metric, it collapses. Deliver X story points per sprint?

WTF? That is not the spirit of agile. That is Big Corp subverting what agile was trying to subvert from big Corp Man Days in the first place.

If a team moves onto a 3rd boilerplate, they've missed the premise of agile retrospectives.

It's really a class problem. Rich people don't want to associate with those filthy poors unless they get to be above them, so they created MBA programs and a managerial class.

I went all the way from Agile maximalist to common sense-ist.

For me agile these days is simply:

- talk to your team to sync often but short

- seek feedback (from stakeholders, ideally users)

- allocate time to work on your backlog so it nice and tidy

and technical stuff:

- release early, release often (i.e. CI/CD/devops stuff)

- automated testing, regular code reviews

- refactoring, working with small commits/PRs

More a common sense, than a "process" or "ideology".

Or this: imagine seeking an estimate on when a house will be built, trimmed, painted, furnished, decorated, and filled with every item needed practically and for comfort by the homeowner.

It absolutely is; look at some of the biggest and most successful software companies in the world. No-one has any idea what facebook will cost or when it will be done (Zuckerberg has said that he was off by at least 5 orders of magnitude IIRC). No-one has any idea what google will cost or when it will be done. No-one has any idea what amazon will cost or when it will be done. No-one has any idea what netflix will cost or when it will be done.

Overshooting deadlines and increasing costs due to materials not being available or them suddenly costing twice as much is normal.

Also coordination between all the various contractors is always getting messed up because the person putting up the scaffolding didn't show up, so the plumber cannot work and cannot come back for 3 weeks, which messes up the plan for the drywall, which gets pushed back, and so on.

They could probably do it more accurately but it would lose them bids, so they knowingly low-ball it. You shouldn’t have that problem inside a company doing software.

Engineers low ball partly because they don’t know what’s involved in a new piece of work. Like walking between two points on a map; they don’t know what they’ll run into on the ground.

Managers and stake holders then bully the engineers into lowering the estimates because even that’s not quick enough. Worse, they scale the team without discussion without realising that will make the project harder to deliver.

As almost everyone knows, construction delays and cost overruns are the norm, not the outlyer. Construction projects here very often suffer dearly from the "2 more weeks and/or a couple more thousand dollars" overrun issues.

In fact, I'm sure at least several people who have been trapped in a home renovation nightmare LOLed deeply after reading your post.

These stories are SO common (and I've seen it in my own family too) that I truly question the sanity of anyone who takes on a home renovation project.

Houses (not housing tracts) tend to be built by much smaller contractors who are in absolutely no position to absorb any setback without going bankrupt, you can't apply the rules for 'big business' to small time operators and expect the exact same outcome.

Constructing resorts also has all kinds of economies of scale that building individual houses does not.

This is very doable in software, and I’ve seen it done for decades from all levels of the org chart.

That's the reality of software development.

The only way you are going to get a hard quote is if you buy a prefab or a house that the contractor has already put up multiple times (and even then there will be some room for adjustment). Oh, and foundations are a completely different story.

By contrast, software development is doing stuff that hasn't been done before. You may be lucky, and have done something similar. If it's really been done before, you would ideally be using something off the shelf.

For me, a project manager who uses this analogy has no place running a software project.

Also, tons of software dev isn't actually about pushing boundaries or doing something that hasn't been done before. In fact, I'd say most development is about implementing known solutions but in a customized way. Just like every other engineering field.

And the project managers who use the analogy don’t make the comparison with significant and novel civil engineering projects. They make the comparison with having their kitchen extended.

When I worked for a larger company with a pretty mature product, this was true - deadlines were internal and arbitrary. It also helped that the product was more of a “nice to have” for our customers than “absolutely crucial” - they didn’t truly NEED anything it did (it was software to make social media marketing more efficient).

But I’m now at a smaller Enterprise SaaS startup, where the product is absolutely crucial to the customer’s work (core operational software for public transportation agencies). The reality is that the product isn’t yet mature enough to close big sales without making sales commits. I dislike sales commits, but we’d legitimately lose every big RFP unless we commit to adding the ~2-5 features they really, truly need that we don’t yet have. And sales commits come with real, external due dates, that you need to be able to meet if you don’t want to piss off your customers. You need to be able to say “yeah, we can build X by date Y”, and then actually deliver on that.

That's just pushing the issue one level higher. Most likely both the customer and you would be happier if you were able to iterate with them rather than building the thing they thought they would want however many months back. (And while it may be harder to set up that kind of contract, it's absolutely possible)

Possible? Sure. Just like it's possible that I could win the national lottery.

In practice, multiple companies bid for contracts, and all of the bids have to be of the form "Deliver $X in $Y months for $Z dollars".

No company is going to accept "Choose us, and we'll work until you're happy with the product or you run out of money, whichever comes first".

Agile doesn't work where there's competition for the project. It can't, because the buyer of that project cannot evaluate "we'll work until your money runs out" against the other bids that state "$X in $Y months for $Z dollars".

Agile works well for internal teams that are sheltered from having to provide competitive bids.

Which still loses to the other bidders who say "$X in $Y months for $Z dollars".

Typically, the intermediate step is to tie some compensation (such as a bonus) to a particular metric, but why not just go whole hog?

“We’ll work collaboratively and evolve the product over time in yet-to-be-determined ways” can work when you have an established relationship and 2-way trust, but there’s not much of that during an RFP. If a competitor commits to their biggest asks, and we don’t, we lose the RFP.

The former is (generally) within the control of the team who is responsible for hitting the date. The latter, (generally) is not.

The process is less important than the people.

this works, but it’s not the only. You can use a mix of buffers and scope flexibility to hit dates too.

My current team only estimates projects not tickets and does not do scrum, and I don’t notice a difference in performance from scrum teams I’ve been on.

Furthermore, I worked on a 18 month project that was estimated by someone who was not on the team that implemented it and it was implemented on time without overtime, and without any special process. Maybe that was unicorn, but it happens.

I think just having a manager/tech lead that cares and maintains priorities is all it takes to stay on track.

Outside of software, you see the same thing over and over again with Lean Management and TPS.

40 years of IT experience lead me to believe that this is only possible for the most trivial of projects with nailed down specifications and where the party building it has relevant domain knowledge. In practice such projects don't really exist. Trivial IT projects are rare, specs change, scope creep is a thing and domain knowledge is built up slowly over time.

What is necessary is that the work does not occupy more time and effort than it strictly requires. That is still a hard problem, but much less hard than to find a way to hit dates that you commit to, unless you give yourself ample room to deal with the realities of complex IT development work.

Note that ALL of these can be summed up in one phrase that I tell my teams often…

TIGHTEN

YOUR

FEEDBACK

LOOP

This is literally the key to life (besides 42) and everything. Want a better relationship with your partner, pet, friends, boss, code, family then tighten the loop.

It could be coming from tests, from CI, from the QA team, from customers, from other engineers in your team, or, failing all that, then yes from management.

But if management is giving feedback that is hard to reconcile with everything else, at this point you can point to CI, the previous tests that were added based on other requirements, feedback you got from customers etc. as justification for your existing actions. It means you're operating with a lot less assumptions because you get data validating what you're doing (and where you should fix things) every step of the way.

The secret lies in the fact that neither giving, nor receiving feedback is free.

Engaging with people, who have different motivations and reasons for being in a job, and stripping away often unhelpful implied titles of rockstar programmer to QA robot, can help everyone to see the big picture and in turn to see how their contributions, their cogs, fit into that picture.

Production-line programming can be left to new automated systems, whilst we can embrace and capitalize on people exercising their intellect for productive, feedback-oriented work.

Making iteration faster let’s you see the results of changes you make faster, that’s the benefit. If you’re iterating faster than you can make meaningful changes then you won’t see any change by definition.

Iteration length is also context dependent. For a multiplayer game I really like being able to see my changes immediately when working on it, I love being able to iterate with other devs on some idea where each iteration is in minutes and we’re chatting about it together but we only test the game itself “as it’s meant to be played” with anyone outside the team once a week. That’s part logistical and partly anything shorter than a week doesn’t usually show enough progress. If there’s some bigger change we’ll also keep our work iterations but pause the larger play tests until it is done.

Short as possible is good advice but you’ve got to look at your own context and make sensible decisions. You can also look at it like following a gradient, if you’re unable to make bigger changes then you’ll just be optimising towards some local maximum.

Micromanagement itself is more about the amount of agency to decide what to do at each iteration.

We need to get back to engineering as an art form, not factory work.

the Agile Industrial Complex realized they can sell that vision to managers, despite it being unachievable.

That promise is so alluring, it’s blinding.

Except you can also measure that we're not hitting any predicted goal times, ever, because our sprint system is incompatible with the nature of our work supporting live systems and simultaneously developing new features. We're regularly taking on work from outside the sprint half an hour after sprint planning, because something exploded and we're the ones who have to fix it.

It's completely impossible, but the vision of predictability has blinded management.

The number of times there's a card on the retro board under "Stop doing" that just says "sprints" on it...

I like Basecamp’s shape up methodology: make bets of efforts that are worth 6 weeks of the team’s time, commit to the best one, timebox it. If it takes longer than 6 weeks, throw it away. This way the risk is limited to six weeks.

Few managers would be willing to stake 6 weeks of a team’s time though.

Well, they think they're minimizing risk, but at best they're just minimizing uncertainty, and the result is predictably low productivity.

- small changes, fixes, additions

- tests, optimizations, refactoring

It’s what you do to move forward iteratively with approaching deadlines.

But to not get stuck in local maxima you need some time off from feedback loops and tactics. You need time to think, design, experiment, sleep. You need time for creative, strategic thinking. Processes, rules, feedback loops, deliverables etc. are all a hindrance to that.

- Kanban (no sprints)

- (45 min) Weekly grooming session

- Thrice weekly stand ups (async)

- (1 hour) Retro every two weeks (sometimes we skip it or it becomes a show-n-tell)

- (30 minute) 1:1’s monthly for seniors, every other week for juniors unless requested more frequently

- Optional (1 hr) themed mob sessions / office hours throughout the week. You bring a problem and the group that shows up will work on it with you. Seniors/IC’s are required to hold office hours at least once a week.

- NO OTHER MEETINGS; engineers are asked to block out their lunch/personal time and refuse any non-essential meetings. They have the right to request meetings be rescheduled to appropriate times or moved async.

- NO MEETINGS OF ANY KIND ON FRIDAYS

Control the amount of Work In Progress by the team.

I’m not sure if that counts as common sense or not but I see it as vitally important.

That’s common sense, otherwise we’re just not taking real-world problems into consideration.

Having worked as a programmer since 1999, I can say that these things were not part of the processes and ideas when I started, at the places I worked. Interpret that any way you want.

I like agile/ scrum/ whatever and what I mean when I invoke those terms, is the list enumerated above.

I dislike the bullshit derivative terms, however. The burn-down charts, estimation poker, story points and all that. (Sure - the poker might trigger a thorough discussion about controversial estimations, so I guess I'll allow it.)

I also really dislike the concept of the daily standup for non-juniors.

I believe all these tools have their place. But they are more frequently used by management to exert power over engineers. “What a dumb take. That feature obviously will not take that long. You are over-cautious as always.” “We had decided that this feature was 2 story points. Why did it take that long?” No, dumb fuck. It was your decision, not mine. I had argued against it, but you didn't listen.

(I'm suggesting that the problem in your process might lie elsewhere - with mapping points to time, maybe).

I like this focused list. I’m always in favour of understanding the need for methodologies yet at times too many people use a methodology name as an excuse and alternative to saying “I want it now, I don’t care how, and I’m not interested in excuses nor if there are problems. But make sure there are no problems.”

Common sense, teamwork and cooperation can help us embrace the salient features of different methodologies without the barriers and no-entry poses sometimes presented by leaders or masters in the name of leading-edge box-ticking, often forgetting that we’re working with people. I’ve not seen many unhappy yet productive, professional and inclusive workers, in any field. Explanations, regular feedback and carrying a team with you can help to reduce stress of everyone in the team.

Processes are key to this, irrespective of the ideology.

This is the one thing that "failed agile" hits against. Large organisations want everyone to use their same crappy Jenkins pipeline. Which means when it keeps breaking then no one can do any work. No one can fix it, and no one is allowed to switch it for a service that isn't down for half a day most days. That isn't "agile".

It derails everything else in the process, what's the point of a retrospective if the most painful parts was because someone has decided outside of the team that you have to use OpenShift for your Kubernetes, but then hires folks who don't know what they are doing and you can't just shift to a managed service until they sort their shit out?

What's the point of standups to get updates if most days people are still stuck on their ticket because the mandated CI pipeline is stuck due to a lack of workers?

I see it in all these complaints about agile, that they don't grasp that they were never given autonomy in the first place. It was always only lip service.

And especially if you change something, make sure you have someway to know if you made it worse and need to go back to what you had before.

Of course it would be dumb not to include all the stuff you mentioned at the start, because we've learned from previous projects that it works. Might as well stand on the shoulders of giants. But it's not the absolute, minimal core.

Personally I'm not convinced that "tighten your feedback loop" is good advice, because I interpret it as "shorter measurement intervals", and the shorter they are the more noise there is in the measurement. But the beauty of iterating the process is that we can try it and see if it works.

They just want to clear their Jira tickets and go home.

Modern agile processes are designed for those people.

This shows the uncanny parallel between Agile and religious sects. If you agiled and failed, you should have agiled harder. Or, even better, part with your hard-earned cash on Agile coaches to show you the one True Path™.

There is only one canonical work, be it a relatively short one, the Agile Manifesto. However, there are multiple interpretations, with each faction claiming they are the one True Path™ to reach enlightenment. Each of these factions has culturally appropriated existing concepts, such as daily stand-ups and backlogs, and claims these are unique to their implementation of Agile.

That leaves you to wonder how developers managed in the dark ages, before the Agile revolution. Did developers chuck out unassigned tickets with their poo buckets on the streets? Could they only deliver Babylonian towers as they lacked a way to communicate with each other daily?

Agile was grounded on solid principles, but is very ideologically naive, which allowed its easy cooption by consultants and management.

Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people:

First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration.

Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc.

The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.

[0] https://www.jerrypournelle.com/reports/jerryp/iron.html

It's also helped me work out a corollary in my mind that has puzzled me for a bit.

My sense, in the Marxist tradition, is that modern organisations are dependent on the extraction of value from highly capable technical resources and, especially outside the tech bubble, they largely resent this dependency.

Let's say developers are an example of a highly capable technical resource, though I am by no means limiting the scope.

This results in a series of mechanics that lead to developers being alienated:

  - From the product of their development (ownership of IP and the resultant value of their code, distance from seeing the positive impacts of their work or talking to those it helps)
  - From the act of developing itself (by its reduction to commercial use and control over how it is done, approval gates, arbitrary coding standards, ticket systems, scrum processes, project managers and product owners, Jira, timesheets etc.)
  - From their fellow workers (stack ranking, power dynamics, labour competition, structural organisational tension)
  - From their human nature and natural talent (by the reduction of their humanity and passion and capability to a mere "developer" or "engineer", use of stereotypes, reduction of humanity to output/LOC/story points delivered, corporate gaslighting at questioning this state of affairs etc.)

My sense of this is that what developers can bring to the modern world is the closest humanity has got to magic. And this dependency is resented. And this resentment leads to workplaces in which this resentment is externalised in the form of debasement (e.g. caricatures and other forms of ego compensation - "they're just the boffins, they don't have people skills!"), control ("the boffins can't really be trusted - better add some process and oversight, and given they don't have the people skills better make sure they aren't anywhere near management/clients!"), and dependency inversion ("sure the boffins can do their coding stuff, but they'd be nothing without us to help babysit and organise things, they don't get the way the world works, clearly it's they who actually need us!"). And this environment in turn leads developers to internalise this systemic resentment as a resentment of themselves, their capability, and their work, aka burnout.

But one question has been bubbling away for me for a while.

How do so many organisations arrive at a system in which it's almost a badge of honour to not be one of the doers? That those who can't do, should, as a moral claim, oversee, and manage, and lead? And we should keep adding more of those people until the doers can't possible do. Even when that produces lower tangible results.

Maybe at one point I internalised the Office Space / IT Crowd idea - the non-doers are "people people" who didn't spend decades at their PCs honing their craft but instead went to wild parties and focused on normal people stuff (the implication of course being that developers are lesser than normal people). Maybe the developers really can't be trusted. Maybe they do need to be managed and watched and distanced. Maybe the code monkey caricature (before being reclaimed by those it was used to demean) is right.

But now I wonder.

What if those people are empowered by the systems into positions of power over developers because in the first instance they affirm the original resentment: what if putting the non-doers in charge safely perpetuates the idea that the developers belong at the bottom of the pyramid and affirms the extraction of their labour in support of the salaries and profits of those above relying on it? Or to put it another way, what if this myth of the code monkey caricature lets the Marxist exploitation and all those incentivised by it justify themselves and the system, and ultimately sleep at night?

And what if the second order effect here is that some, let's call them the senior management class - leaders of the Second order of Pournelle's Iron Law of Bureaucracy in the example above - are more conscious of these dynamics and consciously perpetuating them. What if they are knowingly hiring more non-doers to help keep this balance and control. What if it's not accidental, or people skills, that means non-doers are in charge? What if the very reason they're there in the first place is to be in charge even in positions outside of formal leadership (or at least indirectly support the power of someone else who brought them in for that reason) - after all they're not there to do.

So to close out a long post, a corollary to Pournelle's Iron Law of Bureaucracy: in any bureaucratic organisation the ability to be able to contribute to the goals of the organisation will be an insurmountable barrier to having influence or control within it.

You bet this is the case. Every manager is, consciously or not, empire building. Hiring more underlings so they become more important. This not necessarily mean that they are evil schemers - on average, people are pretty good at convincing themselves that what they do is good and necessary.

Note that Big Corps have much more in common with communism than with capitalism. The way people think about capitalism, where everything is hyper efficient, is incompatible with the huge amount of waste that goes on in big organisations (this is a gross simplification, but you get the point). Communism by its nature means a large centralized bureaucracy, with ironically even worse denial of personal agency and thus more alienation.

I believe this is a holdover from Western society's feudal roots*: the people who do things, especially who do things with their hands, are considered to be lesser beings than those who, by divine right, rule over them.

The ideas and values of feudalism are deeply embedded in our culture, and it's very hard to escape, for instance, the idea of the Good King, ruling benevolently because he is divinely granted better discernment to know what is good and right. (And also the Iron-Fisted Tyrant, which is a seductive fantasy to a different type of person.)

Some of these ideas come to us through the medium of things like just world theory and other Protestant notions, but at their core, they're still basically the same thing: the idea that the people who rule do so because they are Better People, and the people they rule (the ones who have to work for them) are in the position they're in because they are Lesser People.

Thus, rising to management becomes a proof that you, too, were a Better Person all along.

* Not saying other societies don't have feudal roots, just that I'm familiar with Western society

This is the main problem with Marxist theory. It is a well contrived system of beliefs mixing strong emotions, down to earth realism and an overcomplicated and oververbose theorical development that justifies the existence of its own expertise. All of this form a strong opium for most intellectuals.

In the end it loses touch with reality and its scientific pretense very quickly as the predictions don't meaningfully materialize and it fails to deliver on its promises.

History does not follow a direction. The workers never formed a group united just by virtue of being workers. Humans are not malleable not do they have a good natural state. Reforms have always outperformed radical and violent revolutions. Marxist revolutions were never lead, instigated and started by the workers. It miserably fails at deciding on who actuallt gets the power ans make decisions.

A lot of people blame "capitalism" for societal problems but there are no other reasons given except that those are unique to and caused directly by capitalism; capitalism being whatever exists just by virtue of being outside a marxist/socialist society. This is made evident by the fact that marxist revolutionaries are always confronted by the same problems that those who they overthrown had; and so they become labeled as state-capitalists to excuse their faillings.

In a way it's like running a tricky multi-objective optimisation, and the AI will find a way to cheat the stated goal.

In most cases because process improvement suggestions by non-managers (or in the worst cases, by anyone non-external) tend to get ignored by the upper chain or, worse, being stolen by someone along the chain.

Or, to put it shortly, why invest mental energy into improving processes when it's either being discarded anyway or used for the promotion of some manager without delivering any benefit to oneself?

And to make it worse, some organizations are actively set up that way to discourage internal improvement ideas so that the uppermost management is free to hire the infamous Big Four "consulting" agencies to mask what upper management wants as "external and independent" expertise that must now be followed. The Big Four (and their smaller fellows) are a fucking scam set up to enrich the top level at the expense of everyone below.

If you're trying to improve a process just for the recognition, then you're by definition not subject to it, so you're playing the role of a manager. And you excluded that yourself in your first sentence.

Simple: when hired as a developer, I expect being compensated in one way or the other for work that goes beyond scope - especially when it is like in most private companies where you need some form of achievements to show to get a raise or a promotion. And do not forget: when you improve processes, usually your workload will be increased as a result, so in the worst case you'll end up with your manager getting the raise for your suggestion and double the workload as "thanks".

You can fire or train those developers. You can't do that to the CEO.

Agile tranformation requires buy in right to the top. I've seen it work from the top down, never from bottom up and never from middle up/down either.

Agile is a system that ignores how power and status plays work in companies which almost always makes it into a miserable system for everyone involved.

That's why populism is so damn effective on masses who often wouldn't be even able to articulate properly current internal and global political/economical/whatever situation, yet have very strong opinions how these things should be.

I disagree with this statement. Waterfall produced good enough results. What they really want is the most meeting and process heavy approach irrespective of results, because those who advocate for Agile are the people who have no real job and this gives them enough bullshit to bs themselves through every day of their work week.

I'm not a fan of agile either though. I'm not a team player and agile is way too heavy on the framework side for me. But some parts of it are ok like close collaboration with the (internal) client.

Luckily I work for a company that thinks logging our hours in Jira means we're agile so I don't really have a lot to do with it.

There's an additional perhaps even more important aspect contributing here: the sociality of language and shared values in technology as social communities.

IMO If it hadn't been agile it would have been something else; we build common understandings with the terminology we have available, in order for our activities to make sense.

It doesn't matter if the terminology is applied correctly, viz. adhering to the theory (although many escape this accusation by a tacking "hybrid" onto everything). What matters is practical applicability: when we use these terms we're more often understanding rather than misunderstanding each other.

So the terms are used in more abstract ways than intended. (E.g. agile as methodology in health bureaucracy is kinda weird.)