I agree. In my experience that outside visibility is mostly a mirage.

Velocity metrics, burndown charts, point estimations, and all of that have been at best aspirational white lies and at worst outright falsehoods on almost every scrum/agile team I've ever worked on in the past fifteen years.

In actuality many scrum teams are doing something closer to kanban day-to-day under the fake veneer of scrum/agile sprints on top to satisfy management and/or external parties.

Ye padding estimates and report time according to the estimate to make the burn down chart straight was what I learned to do when Scrum was forced on my team for no good reason at all.

-"It is impossible to do accurate estimates"

-"You will get better at it"

I wonder if the Scrum Master knew that in the end "get better" is "starting to cheat".

Do anyone else share this experience?

Estimation often is way off for construction projects, but would you hire a contractor who refused to hazard a guess of how long a job would take or how much it would cost?

Most companies have already hired the contractor though so this little dance is just pointlessness that kills motivation and velocity.

Surely an understanding of how much effort is needed to complete one project or another will inform their decisions of which to pursue.

If scrum provided that it might have some value.

It’s extremely doubtful if you get even a roughly accurate estimate of effort within one project let alone relative effort across two.

Meh. The time it takes to complete a project is usually somewhat flexible. What really matters is how valuable the project is. That is what needs to be estimated explicitly, and not just "I really like these two ideas lets do the cheaper one".

How can you estimate "value" without any idea what resources are needed or what else you will have to forgo to pursue the project?

Value meaning roughly revenue, not profit.

But estimating revenue is just as much of a shot in the dark as estimating effort, and if the effort is great enough it will cancel out the revenue.

The variation in effort is orders of magnitude smaller than that in value. Great value is very rarely canceled out by high effort and low value almost always cancel out low effort.

If revenue from a venture were so predictable you wouldn’t see the Funko Pop company destroying tons of inventory but you do.

You won't gain that understanding until you actually complete both of those projects.

Industries and occupations that manage estimates well do so because bulk of the work is repeatable and predictable, and the people doing that work have done nearly identical kind of work before. There is some variability involved, but there's only so much of it, and it yields to tried and true classical project management techniques (not the things that pass as "project management" in software companies).

For example, even an inexperienced carpenter, asked to install a door system they haven't seen before, will be able to do it two or three times to get a feel and work the kinks out, and then when they're hired to install it in 300 flats of a new apartment complex, they'll be able to give you an accurate estimate of time and effort required.

With software, almost universally, your project is meaningfully unique, and your workers have never worked on something very similar before. The sources of variability are endless - project-unique challenges ahead, programmers never doing that particular component in that particular combination of languages, libraries and frameworks, in their specific versions, programmers and PMs never working in that specific domain before, etc. - and that's before you add stakeholders changing their requirements every other week.

The understanding one needs to have first is that the software project they're holding a stake in is typically best thought of as R&D effort.

You’d think, but wildly inaccurate estimates are not rare at all (and many software projects are fairly routine themselves)

If revenue they bring is estimated you may only need relative comparison.

After all if dev team days that more perspective project is very likely cheaper to make, why would you choose the other one?

See? No need for detailed answers if there is sufficient difference in qualities.

Relative comparison is exactly the scrum method which is why they don't estimate with time. And I don't think revenue estimation is any less subject to variance.

At some point there is a need for time estimates, ye.

I don't ask contractors for a detailed breakdown, just the whole thing. And they either do a high padded fixed price or work per hour.

A friend who is an elictrician might do 100 similar rooms in a building and they have a detailed breakdown of "time per screw" more or less.

So I guess the accuracy depends on the sameness of tasks to do.

Would you hire a mathematician that doesn't "story point" conjectures? Yes. Would you hire one that does?

I find this analogy harder to understand. What am I hiring the mathematician to do in this example?

Research mathematics.

Do you get paid to do research or is there a deliverable at the end?

Yes, you get paid to do research and there are theorems delivered. Just like software projects, some never get delivered.

Unless the contractor is willing to put in writing a "not to exceed" price, I've never put any faith in build estimates. I'd expect a professional to be able to give a rough estimate based on experience and scope, as well as clear communication as the project progresses. Rough estimate meaning accurate withing something like 20-30%, not a detailed breakdown by ever project and material expense.

I try to do the same in software projects. Estimating projects by hours is impossible, fibonnaci numbers are a meaningless abstraction, and spending time each sprint to discuss how it went, what the next sprint plan is and estimate every task is a waste of time.

What I can say is whether a task looks like something that could be done in an afternoon, a few days, a few weeks, or multiple months. Accuracy there comes with experience, but no one gets better by chopping the calendar year into 2 week intervals and pouring a ton of process all over it.

Constructions remains a terrible analogy for software development, as it always has been.

Would you agree to pay for any service where not even an estimate of the price could be given? I guess healthcare is the only example I can think of where anyone consents to that.

Estimates you get are what the estimator thinks you will pay (mostly) not how much it costs them.

Prices in the real world are barely linked to costs. Budget risks are bundled in markets with high variance.

Even to the extent that is true it doesn’t change my argument at all.

Maybe I don't see the relevance of your question then.

Software developers are not negotiating fixed price contracts with their project managers every two weeks. Just how toxic such a situation would be should be obvious. Managers need to understand the context of the job and determine if they're satisfied with the productivity of their employees.

Agile is a development process not a contract pricing strategy. If anything proponents of "Agile" would be more hesitant to quote you a firm fixed price contract than "Waterfall" teams.

They don’t have to hire you every two weeks but they are paying for you and do have to decide how to allocate your time and plan various projects with interlocking dependencies. How can they do that without even a rough idea of how many people it takes for how long to accomplish anything? You are, in effect, still asking them to write you a blank check to accomplish something when they might want to change gears if apprised of how difficult it is.

> How can they do that without even a rough idea of how many people it takes for how long to accomplish anything?

This is not exclusive to scrum.

If you go way back up in this thread the reason given for Scrum being bad is it features estimation, which is useless because it's inaccurate, so telling me that systems other than Scrum feature estimation is not a rebuttal.

This is not really a discussion in good faith.

> the reason given for Scrum being bad is it features estimation

This seems like a strawman and not at all what I was picking up from further up the thread.

---

https://news.ycombinator.com/newsguidelines.html

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

Can you tell me how I was supposed to interpret “it is impossible to do good estimation” as a criticism of Scrum, then?

Research. We’re researching cures to cancer, nuclear fusion, space flight, etc. with no idea what the total cost of ownership will be.

Software dev can resemble research on a smaller scale. The more interesting or complex your requirements and environment, the closer it resembles research.

The original point of agile was to lean into this fact and get out of planning and predicting. Scrum culture has corrupted this goal.

Yeah it turns out if your object is having working software to use to make money you’re not that interested in the mysteries of theory.

Contractors charge you based upon the value they provide you, not their costs and labor. A time estimate is reasonable to expect, though.

Depends. Being billed by the hour is far from unheard of.

Is padding really cheating? I'd describe that as accounting for unforeseen setbacks and correcting for overenthusiastic optimism.

Padding estimates is fine. But not when you report actual worked time, which was what we did to make management happy with the burn down charts straightness.

If you don't report actual time for the task, the excercise is useless.

E.g. shift work time from a slower than estimated task to a faster, or "wait" if the task was done too quickly.

Imo, padding estimates is not cheating. It is just doing estimation right. Teams that do not pad estimates literally always underestimate amount of time needed.

> I wonder if the Scrum Master knew that in the end "get better" is "starting to cheat".

I was a trained Scrum Master and I knew this very well.

What's a trained "scrum master"? I thought the certs don't even have exams.

Totally with you. But since I'm currently preparing to get certified for Scrum, I'd like to point out that all those things you mentioned above aren't really part of Scrum at all...

All the estimation stuff is only for your team, not for outside consumption. If you release it and care what people say about it, then that I don't think that's the methodology's fault.

For the purposes of understanding the way people act, it is important to focus on what they believe to be the truth, not what the truth is.

Software development is incredibly opaque by default (heck, even we don't know how far along we are, most of the time) so anything that improves that will be perceived as improved productivity.

But even if it's not, sometimes progress reporting (however rough) is needed to synchronise with other parts of the organisation. It just needs to be put into a form that is useful (e.g. for agile projects "percent of plan delivered" is not a useful metric) and how it's going to be used should be explained to those producing it. Heck, optimal is if the people making software and the people they need to coordinate with can get together and jointly agree on what type of reporting is actually meaningful for both parties.

Almost always, we're discussing vanity metrics though.

Having predictability and observability is key to a business though, as higher level decision need to be made based on progress of the lower level teams.

Yes, predictability is better for business, but business also needs to accept that there are limits to the amount of predictability you can achieve in software development, and trying to force too much process is only going to predictably increase delivery time. In my experience, working with high level estimates which you keep updating from time to time is the best compromise.

There's no limit to predictability achievable in software development per se. You can have perfectly predictable software development -- if you know exactly what it is you're going to build.

The trouble is -- as with any product development -- we don't know exactly what we are going to build because it depends on fickle market sentiments, new technology, etc. That is what sets the upper limit on predictability, and that can be controlled by high level decisions. (Unfortunately, increased predictability usually means decreased profitability.)

> You can have perfectly predictable software development -- if you know exactly what it is you're going to build.

That's not enough - you can have pretty good predictability if you know exactly what you're going to build, and if you already built the same kind of thing with the same technologies/tools that you're going to use this time. Which is very often not the case: business very often wants new things that others don't already have, or the technologies/tools from previous times are now outdated (or believed so...).

We're saying the same thing. If we haven't built it (or something very like it) before, then we cannot possibly know exactly what it is we are building!

The only way to increase predictability is to pad estimates to the point of meaningless, and management won't accept that. Predictability creates crunch that creates bad choices that waste time and push delivery of working product out further.

> business also needs to accept that there are limits to the amount of predictability you can achieve in software development

Scrum is based around this idea, so adopting it (or adopting it better) should be a sign they understand it. If they start saying "oh points are basically time" then you're probably in trouble.

It’s sadly mostly a sign that they don’t understand it. Most scrum these days is bastardized.

But it has made the PA Consultings of this world a lot of money doing "agile transformations".

In my experience, if the business has a higher level decision that depends on the progress of a lower level team, the only working approach is for the manager in charge of that decision to sit down with a person in the lower level team and discuss the state of things.

A metric compresses a very rich and nuanced opportunity into a false dichotomy.

The problem is even scrum doesn't give the predictability and observability better than kanban. It's extremely hard to predict what we need to do in next 2 weeks and what we can achieve in next 2 weeks. At my current company, half, if not more, of the tickets come in mid-sprint.

So even though the current company is using scrum, in my team I treat it as kanban with story point limited, meaning I start with a certain amount of story points, then add / remove as the sprint is going.

I have seen a very predictable scrum, but the tradeoffs with productivity were comical. In this very large corporation, most developers could do most of the tasks in their 2 week sprints in 3 days of actual work. But they don't know if they are going to get 3 days of actual work, or some other team is going to keep master broken for a week. Therefore, your typical sprint involves mostly fictional standup updates, as to make sure that, when things go more than a little too well, the shameful levels of sandbagging aren't visible in jira or in git.

If you ask me, that level of predictability is a poor tradeoff, but hey, it's the same kind of organization that decides that SAFe scrum is a serious methodology.

SAfE is definitely a bad idea, but keeping master broken should be a "automated tests guard master" retro action.

These type of problems usually are not "actionable" with big legacy apps, poor coverage, etc.

Someone who's not a technical junior needs to make it very clear that breakages will keep happening and support staff will need increasing if no one has the time to figure out a test harness or similar for the software in question. If people are literally blocked because someone else has broken master then they should be working on a test harness right now, in fact. That would actually be useful.

If the sprint commitment gets bounced in favor of stuff coming in mid-sprint, they in itself might be useful information to someone who’s interested.

But the reality is you cannot hope for management to realize it, and for a good reason. If reprioritize a new feature can bring more revenue compared to original plan, or if a better flow is now available due to other team's completed task, or if a severe bug is discovered someday, it's just logical to reprioritize.

That’s not the point though. The point is to have an answer when someone asks why the original thing didn’t get done.

Things changing as a complete 180 mid sprint every single sprint doesn't mean Kanban is more suited as a process. It means your company is not using any process but rather is just a chaotic "who screamed the loudest most recently".

If you do Kanban right you have to have predictability through tasks of roughly equal size. If you will you can think of Kanban as Scrum where all tasks have to either be sized 2 or 3, else they don't make it into a sprint.

New items get added to the end of the backlog by default and stakeholders know when they can expect their request to be done because you know your velocity and queue length.

Putting every single request that comes in at the top of your Kanban backlog is orthogonal and bad.

Scrum is t really designed for an ops team. It's for product dev teams that need to organize and plan their work.

This is for product dev team though, not ops. Don't get me wrong, it's easy to plan works that take less than 7 work days or more than 14 work days, assuming the sprint is 10 work days.

Planning one that's near to 9-11 work days is the hard part.

I think I'd pick any team, not just software, telling me when something will exist over them saying they're more efficient. Efficient isn't even a metric I'd understand in most cases, e.g. a plumber telling me the house will be done as efficiently as possible but I won't know when any particular room will be unavailable so I can't plan around that info.

But isn't the problem exactly that, that oftentimes we cannot know in advance how long it will take to implement X?

I think your parent is trying to say that you won't hire the plumber that only tells you he will be efficient doing his job. But you will hire the plumber that tells you "if I don't find surprises behind the wall, this is a 2 hour job". And then he opens up the wall and finds old corroded iron piping he has to trace down to where it attaches to the main line and replace it all or else you won't get your new bathroom done at all.

And that job takes more than 2 hours. You could also have said "well if that's the case can you poke some exploratory holes in the drywall please so I can get a better estimate" (aka a "spike").

I've seen many people complain about reviews and retrospectives in those threads... But they don't seem like a bad thing to me. The problem is if the sprint is too short. But if the review and retro are done e.g. once every month or so, it can be a good way to see what the other teams have been doing and get an overall picture of your software. And the retro can be a good opportunity to talk about any problems that you encountered, and maybe trigger a solution, or at least just vent about it.

The problem is nothing comes out of it. We run retros and there are action items and they go into a black hole.

The higher ups do it to follow the process and not to fix the actual problems.

It gets tiring after a while of time wasting.

I do not want to talk about these problems again and again and again. I would like to see some improvements.

First few times it works as a psychohygiene, but then it just becomes annoying. So teams stops talking about actual deep issues on those meetings and they are are creating an illusion of happiness.

I don't see retrospectives being that much of a time-sink, it's a 30min/1hr meeting every 2 weeks. Even when I was on a kanban team they still had a fortnightly retrospective, it's useful to have a checkup occasionally.

Every time someone hates Scrum they always complain about 2 week sprints, which are an insane choice (should be 4 weeks), that shows they aren't doing Scrum, they are micromanaging with "Scrum" as the excuse.

Yes, I don't know where the 2 week sprint thing came from, but everyone uses it as a default. I've tried to get it to 3 or 4 weeks but always get resistance. The main reason I think is the business wants their features NOW.

Going longer than two weeks for a sprint just makes the lies happen at different place. There are approximately 0 teams that can accurately state all the work they will need to do four weeks in advance, even two is a stretch. Planning and retro per sprint become a burden though, on less than two week sprints.

I currently work at a place that does 1 week sprints. It's bad.

This is just doing scrum badly... management break it by insisting the metrics etc are used as a reporting mechanism, they just cannot get away from "intensive control mode". The metrics are supposed to be for the team to use internally.

But I do agree with the post - Kanban is way way better and you can include the scrum continuous improvement elements as well on a regular cycle.

> The metrics are supposed to be for the team to use internally.

You've already pointed out that this is basically unheard of but I'm struggling to think of situations where scrum metrics have provided benefits that outweighed the cost of taking, and exposing them.

My experience has been that most development teams intuitively recognise problems that might be revealed by these metrics. Is this not often the case?

This is where a decent scrum master or agile coach is valuable - fighting off the misuse of the metrics by external (to the team) management.

The thing is with metrics is that they are hard to argue with, whereas intuition is a debatable opinion. Perhaps not everyone in the team agrees. Perhaps it's about showing interfering management or dependent teams where the issues lie. Perhaps the team doesn't realise how long a particular stage in their workflow is holding things up - I mean they know that stage is a problem but have they realised it's actually their biggest blocker to finishing work?

Also depending on what you use to generate metrics - something like ActionableAgile that use Monte Carlo simulations against a few weeks worth of activities can show a team what their cycle time is, what items are in danger of blowing that and how long it's likely to finish just their current backlog (for example).

As you say, in their worst case they tell teams what they already know and then again that's the value of a good SM or AC - finding out why nothing is being done about it (whilst beating away interfering managers).

>The time taken to do team task estimation, update jira tickets and have retrospectives is time spent not programming.

Agree with everything _expect_ retros.

I think the value of the retros is proportional to how senior the team is and to how seasoned managers are. But for me retros are extremely high value _if_ you have:

- Teams that can look at problems head on and be civil addressing things without making it personal.

- Managers that have a 'clear roadblocks' mindset and have a service-providing mindset towards the team.

In this scenario you can really have high value retros, with continuous adjustments to workflow, processes, design, without continuously repeating the same mistakes or falling in the same pitfalls.

I found that retros tend to devolve into gripe sessions when the team isn't able to change its environment. Best to keep them focused on things they can actually change, and just to let them fix what they need instead of talking about it.

Also, after an initial stabilisation period, teams need fewer and fewer changes. Talking about it every sprint gets really pointless and takes time away from the things they're good at and they're motivated to do.

Proposing retro board be available 24/7 and retros be skipped if board is empty is good use of retro time. ;)

This sounds amazing.

But I do wonder if it just encourages people not to bring up real problems so that they can skip a meeting.

I think having the board up at all times is a great idea though. Just have to have some way to figure out "is it empty because things are fine or is it empty because no one cares anymore".

I've worked on a team that had amazing retros: lots of difficult discussions that resulted in change. We also made sure to discuss positive things too to ensure they keep happening. I think a lot of people miss this about retro.

I also worked on a team with terrible retros. The "good" column would always be extremely long and was only full of platitudes, nothing of actual substance that was brought forward into the next week. If anything difficult came up, someone would say, "Ok, let's book a meeting to talk about this later". It was an extreme misunderstanding of the value of retros. Though everyone else seemed to enjoy them so that's arguably valuable? I dunno.

Anyway, yes, lots has been said on this topic in the past. SCRUM is just Agile for stakeholders. There is the whole "SCRUM But" joke but I actually think a bunch of things in SCRUM are valuable BUT if something isn't working out, don't do it! I get the impression that lots of companies just do all the rituals "because SCRUM" and don't actually understand why they are doing them. Case in point, I was at a company that would always spend time story-pointing but no one ever looked at them. Not the teams and not the stakeholders.

> Especially if nobody can actually quantitatively measure the loss of efficiency.

Yet ironically they care and tout about their ability to measuring efficiency.

An easy way to boost efficiency: hide inefficiency out from view!

> most management teams would prefer their employees to be predictable and observable (ie, under control) than be efficient.

we're in this exact phase - sacrificing efficiency, team morale and independence to make the boards look cleaner to "raise the bar" per se.

We're a small efficient team in a massive org. Manager does a great job of not making stupid decisions, values people's time and is clear about the metrics the team gets evaluated on.

somewhere along the way he's, missed the reality that clean user stories and tasks, complete with points down to hours and acceptance criteria can still be completely f$#king wrong compared to the actual work that needs to be done. Not because the people are idiots but because thats how most software work is. You find out when you actually start working on it.

But if you mandate that you need a well cut plan before code gets written, you mostly get well written $hit.

We just embarked on this journey which I can mostly predict the outcome of.

sad, true.