But in a case like yours if sounds like steps 2 through 5 aren't strictly relevant to the change you're making, so it's no wonder that your estimate is blown out the water. (Sorry if I'm wrong, you've obviously got more context on your own platform than I do!)

It does make me wonder though if there is a trend of 'hiding' necessary maintenance work in larger tasks - if so that seems to me to be indicative of a larger problem around not being given/making the time to do those jobs as their own tasks.

Yes, there is. Probably the best four projects I worked on, there was collusion to spread out the cost of necessary maintenance across all work items. Because nobody will agree to doing maintenance stories (this is how Scrum hijacks developer ethics). Never ask permission to do something that must be done.

Or to look at it another way, if no stories are maintenance stories, then all stories are maintenance stories.

I used to be all about jumping on the next fancy thing as soon as it was (pre)released. Having been burnt and learned my lesson, now I delay until it's blindingly obvious it needs to be done.

But yeah, if it needs t be done, do it.

Version x.y.0 of most software is "public beta" quality. x.y.1 is "release candidate". x.y.2 might be production-ready.

You need to get an agreement with the team that you're going to amortize future upgrade treadmill activities across all work that is done. You can't mortgage future productivity to get short term gains when your project is supposed to run for 5, 10, or 20 years. You'll spend most of your time working slowly if you work that way.

yours and a lot of other comments seem to hint that this work may not be strictly required? I'm a bit surprised.

At least in my case, as a public facing SaaS I don't think running Node 6.x past it's LTS window is a valid choice, so better to bite the bullet now. If I went about doing the minimum work, it just adds more code/complexity that needs to be refactored when upgrading Node.

People are hinting that it is not strictly required to get what was initially desired done.

And that seems 100% true.

You do not have to do everything at one time. You can easily to one thing, and then right after that, literally right after, do another.

You could have gotten the proxy up and going, and then upgraded node.

It's not about "doing the minimum work" it's about understanding the problem space, what can be split up, and how to attack the surface area efficiently.

You're also introducing other problems that can occur by doing multiple things at the same time.

Generally speaking I advocate my developers to be really good at separating problem spaces and attacking accordingly. It's really hard to take on multiple issues at one time and actually have high confidence in not breaking many systems along the way.

okay thanks for the tips, you and everyone else :)

No, it has to be done. But some people won't believe it until they've had a couple tough upgrades that show the cost of laxity.

On one project there were CERT advisories out, and we had deferred upgrades due to breaking changes in those upgrades. All of a sudden we had to deal with the upgrade and a security issue at the same time. It was ugly. After the second time, we started putting at least one upgrade story per month on the work queue. Sometimes we let the engineer pick what they wanted to upgrade (just upgrade something!). Other times we picked the engineer for the work that needed to be done.

You either need to upgrade from Node 6 or you don't.

That's nothing to do with adding a new feature, and should be planned accordingly. It's unfortunately very common for developers to hide maintenance tasks in other work, but clearly not ideal.

I agree. However, to be devil's advocate - quite a lot of developer teams aren't allowed to migrate to new systems until way too late, so it's not surprising they do it in unrelated feature requests/bug fixes.

While all platforms have some measure of this phenomenon, the update treadmill problem does seem to be especially acute in the javascript ecosystem.

I update Java dependencies maybe once a year; it's usually painless and incompatibilities are highlighted by the type system. If you let your JS dependences get stale by a few months, you're looking at hours or days of work. And that doesn't count the massive shifts in build systems and frameworks that seem to happen annually.

I've spent the last 5 years on Node, and honestly most modules are fine. The only ones I've had serious problems with are those created by Google. They keep taking huge complex dependencies on things like Grpc that break between Node versions, and keep churning their other library signatures (or deprecating entire modules).

Maybe in the last year they've settled down, maybe not. I learned my lesson and don't take dependencies on them anymore. Much easier.

I just spend maybe half a day a month updating everything to the latest and testing. There are the occasional gotchas, but it's not so bad... When you leave a codebase to rot for a few years, it doesn't surprise me that it's more painful.

My own 'magnification' index is as such:

Take whatever time I think it should take. Then double that number and go up one time unit.

Exp: I think it should take me 3 hours this afternoon to do this -> 6 days.

I think it should take me 2 days to to this -> 4 weeks.

I think it should take me 1 week to do this -> 2 months.

I think this should take us one month to accomplish -> 2 years.

Such approach assumes that literally all tasks are minefields full of unpredictability. But are they really? In my experience such reasoning and 'magnification' applies only to minority of cases, while most of the time you hopefully deal with well-enough understood domain and codebase to accurately estimate the effort based on the initial judgement.

For me, it's more of a guestimating rule. Sure, some tasks take less than I'd expect them too. Sure, some take a LOT longer than even my guesstimate. But this helps me 'plan for drought and pray for rain', so to speak.

To clarify why this strategy is not as bad or egregious as it seems, one must only ask: what are the consequences of overestimated time for work (that gets delivered early?) And what are the consequences of underestimating work?

Would you rather deliver a product with finished surfaces or cut corners? Proper design and good overall test coverage, or zero budget for maintenance?

It also helps to understand that management will ultimately decide that we need to do the project in half the time, and with these extra features you didn't ever plan on building.

My budgeting strategy is to always assume that for whatever scope you have planned, 50% of the complexity that you will need to deliver in order to reach the finish line, remains unknown at any given stage of planning. Compensate upward if there still remain known unknowns.

The consequence of overestimating is you don’t get the project. Occasionally that’s the consequence of proper estimation too, as people tend to underestimate when selling the project to get it through. Once I estimated a project at 3 months team of 5. They told me it should be 3 months team of 2. “We are doing sales mate” they told me. “You are, I am doing delivery” I responded, and then did neither sales nor delivery. ..

Estimates for internal client is a different ballgame than estimates for sales, it would seem.

My approach to the same project would be different if I know that I get to walk away when the project is over, versus if I know I'll be the one to maintain the project when this phase is over and it goes into production. I work for internal clients lately, and I can't honestly say it's my way or the highway, but if you ask for an estimate, I'm going to give you an estimate.

If you tell me the deadline then we're having a different conversation entirely. (And that's OK! Deadlines are better than estimates in my book, especially if the consequence of missing the deadline is some unfinished garbage goes to production, or doesn't ship at all as we had to move on to the next thing. If you asked me for an estimate and told me "that's too much, do it in less" then you didn't really want an estimate, did you? Just tell me the deadline and I'll deal with it, in that case, let's not play games.)

Here's another approach: tell me how much time you want to spend on this project, then I'll put a pen to paper and tell you what part of the scope we can deliver given the resources you've allocated. Don't like that very much? OK, you write the plan and I'll say "yes sir." But that's not why you pay me, is it?

Estimates are a function of scope, time, and cost. If you tell me to do it in less time, I'll do it in less time, but it's going to come at the expense of one of either feature scope or quality. This is not a negotiating tactic, it's just a statement of fact. If you tell me I have to shave off a bunch of time without sacrificing anything, then we're not having an honest conversation and we're not going to have a good time. I'm sorry you had a bad experience, mate.

For me it works pretty well if I imagine someone else (another team member who is less experienced) doing the job, and myself reviewing the code. It turns out my estimates about other people's work are much more accurate... :)

This is brilliant.