News for hackers. Stuff that matters.
Re:The ‘flawed five’ engineering productivity metrics by yodon on Monday May 16, 2022 @12:51PM (#31400142)
My favorite quantitative metrics for engineering teams:

  - Avg time from code review requested to code review picked up
  - Avg time to complete code review
  - Avg time from eng done to first customer using it
  - Avg time from eng done to full production release
  - Fraction of tasks started that never reach a customer
These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics world via the acronym TIM WOOD (or TIM WOODS)[0] and are similar but not identical to the DORA metrics.

The time to complete a code review is there (for example) not to focus on the amount of time actually spent performing the code review but to focus on all the waiting around the actual code review, which is typically much much longer than the time spent doing the review itself. It's not uncommon to see organizations where an engineer will submit code for review and then have to wait a day or more for someone to pick up their request and then another day or more for that other person to get around to reviewing it. If there are comments on the commit that need to be responded to, you can see additional delays. These "minor inefficiencies" can have huge impacts on the poor dev who is trying to get their code merged, and cumulatively they result in significant increases in feature latency, the total calendar time required to ship a feature.

[0] https://www.shmula.com/28695-2/28695/

Re:The ‘flawed five’ engineering productivity metrics by feoren on Monday May 16, 2022 @02:03PM (#31401181)
> - Avg time to complete code review

Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.

Re:The ‘flawed five’ engineering productivity metrics by kqr on Monday May 16, 2022 @02:34PM (#31401546)
Have you measured this in your organisation? In my experience, every team has a fairly distinct cut-off point where if you make PRs larger than that, they start to hang around in someone's review queue forever.

Every time I've made a larger PR I've regretted it and subsequently split it up into smaller ones. I get higher quality feedback and faster feedback to boot that way.

Re:The ‘flawed five’ engineering productivity metrics by dkersten on Monday May 16, 2022 @03:54PM (#31402437)
> if you make PRs larger than that, they start to hang around in someone's review queue forever.

Or you just get a "ship it" without more than a cursory glance.

Re:The ‘flawed five’ engineering productivity metrics by yodon on Monday May 16, 2022 @02:14PM (#31401308)
> Doesn't [Code Review time] depend heavily on the size of each PR?

The amount of active "eyeballs on code" time to complete a code review can of course vary tremendously. That said, when I see code reviews taking a week to complete, only very very very rarely is that a sign that developers are routinely spending a solid week of eyeballs-on-code time completing those code reviews. Most of the time it involves a relatively small amount of focused review time and a huge amount of not actually getting around to starting doing the code review time.

If everything in your shop is so well handled that you're not wasting any time waiting for code to get reviewed, in practice your shop is probably also already making good choices about how to structure code for efficient reviews as well (but that's more an anecdotal observation about correlations, not directly an observation about causality).

Re:The ‘flawed five’ engineering productivity metrics by imoverclocked on Monday May 16, 2022 @04:23PM (#31402742)
Just starting the review can take a while. However, engineers who might be reviewing code instead of other activities may be providing more value/higher velocity/better quality with those other tasks than breaking to just review a PR.

Someone once argued to me that each feature PR should have very few surprises if the feature has been planned and communicated well with affected parties before implementation even begins. The opposite is often true in some organizations.

Re:The ‘flawed five’ engineering productivity metrics by jrockway on Monday May 16, 2022 @06:11PM (#31403744)
Personally, if I get a small PR, I review it in about 30 seconds. If I get a giant PR, I review it in a matter of hours. You are usually aware that such things are coming; that's why people provide daily-ish status updates and that sort of thing.

I have also found that high review latency begets large code reviews. It's kind of a vicious feedback loop. If you always review PRs immediately, people are less afraid of splitting up their work, and PRs will become smaller. (It's hard to "stack" reviews, so people avoid it. That means that 3 PRs become 1 PR, so they can be mentally done with the task and start something brand new while waiting a week for your code review. The alternative is hoping that the bottom of your stack is approved without any requests to change anything, but what happens is that large changes are requested and the two floating PRs are now impossible to merge back in, or are irrelevant. If the bottom of the stack could be reviewed in an hour, then all that work wouldn't have been wasted, and it reinforces the good habit of doing 1 small thing at a time.)

I look at all of this as a systemic issue, and not a personal issue. The team needs to set an aggressive SLA, guided by what the engineer is expected to do while waiting for a review. Then engineers need to treat the objective seriously; code review is as important as any code you're writing. That's why you're part of a team, and aren't a 1 person army (which is a totally viable approach; I write a ton of code in my free time by myself, but I don't necessarily think that that's how I should behave at work.)

Personally, I have Github's Slack notifications on, and almost always start the review as soon as I get the notification. I know that the person waiting for the review has no work to do other than to build on top of this foundation; the less they do without feedback, the better. (Plus, they want to get their code Out There, that's why they wrote it.) I don't think your organization has to be that aggressive, but I'd aim closer to "hard interrupt" than "within 1-2 business days". (No, I don't review PRs that come in at 7PM on Friday night. Those wait until Monday morning.)

The other metrics listed a few comments above are really a function of organizational policies. Make your releases easy, so you do them often. Make sure that your PMs know what users want, so you don't implement code and throw it away. Make sure that CI is fast, so there isn't a delay in pushing to production. And make sure your process is realistic. My team used to spend a lot of time manually testing stuff in staging before releasing to production. It never found any issues that the test suite didn't find, and we prefer the automated tests anyway, so eventually we just made every commit to master a deployment to production. Click the "merge" button and it's out to the users. No unusual number of surprise production outages, just a vastly reduced time from idea to being in the hands of the customer. (Code reviews ended up being the limiting factor. People really don't like doing them!)

Re:The ‘flawed five’ engineering productivity metrics by mandeepj on Monday May 16, 2022 @09:32PM (#31405033)
“Draft PRs” for the win
Re:The ‘flawed five’ engineering productivity metrics by mdoms on Monday May 16, 2022 @03:01PM (#31401847)
Large PRs are one of the most serious anti patterns in day to day work.
Re:The ‘flawed five’ engineering productivity metrics by puchatek on Monday May 16, 2022 @02:52PM (#31401750)
Just the fact that these are team-oriented and not individual-oriented makes them above average valuable already
Re:The ‘flawed five’ engineering productivity metrics by kqr on Monday May 16, 2022 @01:13PM (#31400462)
I use a similar set:

- Time between idea first recorded and idea in production.

- Time between idea merged to main branch and idea in production (in some places this is negligible, in others where ideas spend most of their time hanging out -- done, but collecting dust).

- Time actively spent on task over the time between work started on idea and idea merged to main branch. (This is "flow efficiency" in lean terminology.)

Other than that I use the DORA metrics too. I measure defect rate by looking at proportion of merge requests that fix defects, because mostly that is a stable distribution.

Re:The ‘flawed five’ engineering productivity metrics by kqr on Monday May 16, 2022 @01:09PM (#31400405)
These are great!

How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable.

In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to use it.

I would even say that canceling many started tasks is directly correlated with a quick cycle time, by Little's law.

Re:The ‘flawed five’ engineering productivity metrics by yodon on Monday May 16, 2022 @01:26PM (#31400637)
> Why are you using the last one?

The last one is definitely in a different bucket for me than the first four. For starters, all of these are there primarily to encourage conversation. That said, the first four can make a lot more sense to try to graph and track and optimize. The last one tends to be more purely about driving a conversation.

At the top level, if you're genuinely doing a ton of learning along the way to shipping the right feature to the customer, then arguably that value is ultimately reaching the customer. If on the other hand you can't decide what the goal is and you keep changing your mind (as is often the case), then you tend to end up with a lot of dev investment made in things that simply never ship.

It's also worth differentiating technical "spikes" from feature "experiments." In my vocabulary, spikes are things where you're internally assessing a question like "could we do this" and experiments are things where you're externally assessing "do customers want this/does this have the impact we want." If you have a lot of experiments that don't reach customers, you're burning a lot of dev time on things that aren't actually experiments (because by this definition experiments need to reach the customer surface to deliver data). That's a signal you should probably be looking at. Spikes generally only reach the customer indirectly (through an eventual shipping feature), but if you have a lot of spikes that don't ever reach the customer in any way that's also a signal you should probably be looking at.

Re:The ‘flawed five’ engineering productivity metrics by district487 on Monday May 16, 2022 @01:30PM (#31400719)
not the OP, but I would use it as a leading indicator that there is something wrong with the way we're scoping and prioritizing tasks. If 100% of our tasks are no longer viable, it's a signal that we're doing something fundamentally wrong earlier in the chain, e.g. are we misunderstanding the user problems? the market? the wrong prioritization? not enough scoping done?
Re:The ‘flawed five’ engineering productivity metrics by Brian_K_White on Monday May 16, 2022 @01:11PM (#31400440)
What do I care about start to finish time?

The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently.

A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress.

It's fundamentally not a synchronous process and I say a mistake to try to make it one.

Edit to address a sense I'm getting from several comments at once, that I may be a manager who doesn't understand or care about developers:

I AM the developer (& sysadmin) with a bunch of fresh and stale projects and it doesn't bother me at all. I always have work to do but it's not stressfull because that is about your boss(es) not about the number of open items. If anyone tried to say all that stuff needs to be done yesterday, sure that would be a problem. But all they really are is a spectrum of priorities. Some are just ideas little more than stubs for possible future interest, some would be nice but may never be justifiable in a strict easily quanifiable sense, you can't bill any customer for the hours, and yet would still be nice, would be one of the things that sets your product apart and attracts the customer in the first place, and some have normal priority, and when something high priority comes along, it simply displaces the rest. It just requires a boss who doesn't say that every single new item is always the most important.

That stack of available projects, and the age of half of them, doesn't bother me at all. Half the time, a different project is as good as a walk in the park for the reset / fresh look factor when stuck on something.

Re:The ‘flawed five’ engineering productivity metrics by kqr on Monday May 16, 2022 @01:21PM (#31400570)
This is a common conflict!

One perspective says that idle engineers cost money, so you should load up your engineers to 99 % utilisation with a huge backlog of tasks. The consequence of this is that tasks take, on average, months to complete because they sit idle most of the time, not being worked on.

The other perspective is that idle tasks is what's expensive. Thus you must keep your engineers lightly loaded (say 70 %) so they can, in your words, pounce on new tasks. With this setup, most tasks spend a negligible time waiting and most time actively worked on.

Now, how can idle tasks be more expensive than idle engineers? There are whole books dedicated to explaining this. I'd recommend starting with Reinertsen's Principles of Product Development Flow.

Re:The ‘flawed five’ engineering productivity metrics by thraxil on Tuesday May 17, 2022 @08:53AM (#31409785)
It's always kind of amazing that so many engineers (and engineers turned manager) have encountered enough queuing theory to understand that if they keep a CPU at 99% utilization, it will have a detrimental effect on task latency, but can't understand how to apply that same reasoning to engineer utilization.
Re:The ‘flawed five’ engineering productivity metrics by Brian_K_White on Wednesday May 18, 2022 @07:48PM (#31429002)
An engineer knows where analogies apply and where they don't.

My pool of available work is not impacting whatever I happen to be working on at the moment. I am not doing 1 second of work on 60 different projects every minute.

Re:The ‘flawed five’ engineering productivity metrics by lordnacho on Monday May 16, 2022 @01:27PM (#31400663)
You need idle time as an engineer. Otherwise if a higher value item appears in the pipeline after a lower value item you'll be working on the lower value thing first.

Idle time is also where you do useful things like checking for updated dependencies, running a linter, generally reading up on the tech you're using, and making the code ready for future edits. Not so idle at all but not really the same as working on a ticket either.

Re:The ‘flawed five’ engineering productivity metrics by the_jeremy on Monday May 16, 2022 @04:09PM (#31402596)
Do you work somewhere that gives you this idle time? We explicitly have tickets for any tech debt or dependency updates we need to do, because otherwise we don't get time to do that.
Re:The ‘flawed five’ engineering productivity metrics by lordnacho on Tuesday May 17, 2022 @02:28AM (#31406711)
I decide my own work so yeah. I realise not everyone gets that luxury.
Re:The ‘flawed five’ engineering productivity metrics by dcow on Monday May 16, 2022 @01:22PM (#31400586)
Some developers are very much not efficient in a "fully pipelined" stage. Lots of context switching can slow down everything in the pipeline. Even if you're waiting on a code review and burn a few hours, it can still be more efficient to stay focused on driving the task at hand to completion than trying to spin up a small task during that time and switching back to respond to review comments every so often.
Re:The ‘flawed five’ engineering productivity metrics by Brian_K_White on Monday May 16, 2022 @03:17PM (#31402042)
I take my idle time whenever I want, either by simply not working right then, or working on something low priority that I just want to work on. The size of my pool of available projects has no bearing on that positive or negative.
Re:The ‘flawed five’ engineering productivity metrics by dilyevsky on Monday May 16, 2022 @01:56PM (#31401095)
So just rubberstamp your pr with a nit to score awesome on the first two?
Re:The ‘flawed five’ engineering productivity metrics by yodon on Monday May 16, 2022 @02:04PM (#31401191)
> So just rubberstamp your pr with a nit to score awesome on the first two?

If you're saying you can CR your own commits, that's a different problem to discuss unrelated to the numbers.

If you're saying you're coordinating so closely with the other devs on your team that they're picking up your CR's immediately and they're able to legitimately approve them with just a trivial nit, then congrats - you're "gaming" the system in exactly the way it's designed to be "gamed." In general, if devs are talking to each other and aware of what each other is doing, efficiencies are high and latencies are short. If no one is talking to each other, efficiencies are low and latencies are high. Talking to each other and coordinating with each other is a good thing.

Re:The ‘flawed five’ engineering productivity metrics by dilyevsky on Monday May 16, 2022 @03:16PM (#31402024)
So you’re becoming “aware of what each other is doing” by half reading the pr title and going straight to approve button? I think “legitimately “ is doing a lot of lifting here too
Re:The ‘flawed five’ engineering productivity metrics by yodon on Monday May 16, 2022 @03:49PM (#31402374)
You seem to presuppose that the people involved in the team conversations about these metrics are stupid or clueless to a point of willfulness or intentionally bad actors. If that's the case, as with the case where you're somehow able to approve your own CR's, you face problems unrelated to the numbers. If those are the kind of problems you face, you need to fix them well before you try to bring in any processes designed to help a well functioning organization function better, including the case you called out where devs are approving code reviews without understanding them.
Re:The ‘flawed five’ engineering productivity metrics by a1369209993 on Monday May 16, 2022 @09:25PM (#31404974)
Actually, they seem to be presupposing that you're using these metrics as a target to judge employee performance by (in the style of key productivity indicators), rather than as a diagnosic tool to debug your/management's own mistakes/potential improvements in how the team is organized, and then pointing out in what way the former approach - which always fails regardless of which metrics you use due to Goodhart's law - fails with this particular choice of metrics.

Edit: granted, sometimes the result of said debugging is that it really is a hardware problem, and with how unstandardized people are, that's actually common enough that you'll likely run into it. But you don't fix (for example) a disk drive that reports writes as complete before they're peristed by stack-ranking drives and trying to pick the ones that lose data less; you fix it either by replacing the faulty drive immediately (assuming you can get working drives), or (more relevantly to dealing with people) by working around it in software.

0: http://en.wikipedia.org/wiki/Goodhart's_law

Re:The ‘flawed five’ engineering productivity metrics by dilyevsky on Monday May 16, 2022 @06:09PM (#31403723)
Never said anything about approving your own prs nor bad actors. It’s just basic human psychology especially when money is involved and if you’re going to pretend it does not exist you got another thing coming
Re:The ‘flawed five’ engineering productivity metrics by ajchida on Monday May 16, 2022 @11:16PM (#31405689)
If you also measure the defects in production post release, we can co-relate the quality of CR and narrow down the bad actors
Re:The ‘flawed five’ engineering productivity metrics by dilyevsky on Tuesday May 17, 2022 @02:11AM (#31406607)
Imho if your intent for code review is to find bugs you’re doing it wrong.
Re:The ‘flawed five’ engineering productivity metrics by hatware on Monday May 16, 2022 @02:00PM (#31401150)
You can game any system. I think the point is to give yourself guardrails that help you make good decisions for the present and future.
Re:The ‘flawed five’ engineering productivity metrics by dilyevsky on Monday May 16, 2022 @03:14PM (#31402005)
Wouldn’t even require bad actors - people will naturally er on the side of stamp and forget if you announce this metric as goal
Re:The ‘flawed five’ engineering productivity metrics by marcosdumay on Monday May 16, 2022 @03:05PM (#31401910)
> Fraction of tasks started that never reach a customer

If you optimize this down, you punish any kind of innovative or ambitious task.

Re:The ‘flawed five’ engineering productivity metrics by lukeramsden on Monday May 16, 2022 @03:32PM (#31402209)
Depends how you measure "reaching", i.e. all refactorings and optimisations all reach the customer technically "reach" the customer as soon as they're deployed (essentially, anything that meets its definition-of-done is "delivered). I think the wider idea is to measure work being interrupted and abandoned, or being abandoned due to not being necessary, both of which could be dubbed signs of inefficiency.
Re:The ‘flawed five’ engineering productivity metrics by marcosdumay on Monday May 16, 2022 @03:54PM (#31402440)
Hum, no. I'm talking exactly about abandoned work.

If you don't abandon work, you either don't take any risky task, or you create a really shitty environment and will lose effectiveness soon because of it anyway.

And yes, for some software you don't actually need to take any risky task. At least for a while. But if you are in a position to set that policy, you are very likely not in a position to have a clear enough view to know if your software is such one.

Re:The ‘flawed five’ engineering productivity metrics by jrockway on Monday May 16, 2022 @06:28PM (#31403869)
This is worth a read: https://apenwarr.ca/log/20171213

Specifically, look at the "PM can't make up their mind" graphs (https://apenwarr.ca/log/20171213#slide13). That's the state that you're trying to avoid.

I think it's fine to have tasks like "investigate the feasibility of X" and finding it infeasible. To me, that outcome is similar to delivering a feature (your customers will pay less for that than an actual feature, though, probably). What you want to avoid is "do X at any cost", and then the first steps are "integrate with third party service Y" "get a support contract for service Y" "do a giant refactoring to better use service Y", and then finding that X is actually infeasible. Now you have complexity, cost, and wasted months without having anything to give to the user.

(Conflict of interest note: go ahead and do that. More work that never ships = more developers needed = higher pay for developers!)

Re:The ‘flawed five’ engineering productivity metrics by yodon on Monday May 16, 2022 @04:13PM (#31402644)
In practice I generally see the abandoned work fraction dominated by much less interesting and much less valuable poor planning decisions, false starts, and misalignment between teams. Genuine high-value risk taking is typically already so rare in most organizations that it doesn't even show up significantly in a count like this. Equally importantly, when those genuine risk taking activities are terminated they typically happen in the form of an experiment that reached a customer at some level as part of the decision to terminate the effort. If it reached a customer in some manner like that it's not in the bucket of tasks started that never reached a customer.
Re:The ‘flawed five’ engineering productivity metrics by splittingTimes on Monday May 16, 2022 @08:33PM (#31404676)
They sure are nice, but how do you collect that data in an automated fashion?
Re:The ‘flawed five’ engineering productivity metrics by rsanek on Monday May 16, 2022 @11:05PM (#31405624)
Top two should be available through an ETL of whatever VCS is used. Next two depend on how fine-grained you want to be--either you don't automate because you stay high-level on the order of products, or you go lower-level and use feature flags or even basic logging for when a given feature is used. Not sure how you easily measure the last one though.
Re:The ‘flawed five’ engineering productivity metrics by kqr on Tuesday May 17, 2022 @12:42AM (#31406124)
Obvious follow-up: why do you need to automate it? You get decent precision by sampling just 10 % of what's going on. Depending on team size, that's manageable by just one person spending half an hour on it here and there.

And in the process of asking questions and poking around this person learns a lot about what's happening.