Productivity is not a cut-n-dry stat for any kind of knowledge work.

For an admin, we could say they were more productive if they handled more cases than last month. But this does not account for tricky cases or ambiguous cases, which might have taken 2x time to sort out but on rote numbers it looks they were less productive.

For us, there are no specific goals(except arbitrary imaginary deadlines by clueless management and sprint points) to correctly measure productivity. A junior engineer may spend 4h each day blasting out many lines of codes, a mid/senior may spend more hours in reviews or meetings. A 5 line PR with test coverage is faster to review than a 800+ LOC PR with additional tests and have significant risk of breaking something, so review number is also not a good indicator. I need to coordinate with min. 12 people to get any new credentials or access to specific env owned by another team, which is not countable as productive but unavoidable. How about that garbage meeting where I was just spending time yawning because some new tech lead who believes that a modular monolith system is better if it was rewritten as event based. Productivity measurement is hard.

To take your envy(I sense it on the quoted remark) to another place, think about how you go about measuring productivity of a CEO, PO, PM, Scrum Master, Agilist/Agile-Specialist etc?

For individual, productivity is measured on multiple axis - new long-term features, throw-away and prototypes, code maintenance, fire-fighting, code reviews, high-level design, outside-of-team communications, intra-team communications (for example helping teammates), archeology, specific parts of your project, probably others. In ideal team, everyone is strong on at least one or two axes, so no person is worse than others. In real teams, I've seen people who were weak on every single axis and did not seem to improve with time. I've also seen people who are great at everything, and it's extremely nice when such people exist - but they are very rare.

(Btw, re your specific example: if you weren't productive because you were waiting for others to approve your access then you were not productive. There is nothing complex about this, there were plenty of times when I've said, "I did nothing for project S last week because they still could not give me access")

With respect, that goal is far from clear. How do you measure how much they like the product? Have we achieved more "likes" this sprint than last?

Customers like a product because it removes their problems. QED.

Completely agree: focus groups, A/B testing, Net Promoter Score, 5-stars, like/dislike -- all of those systems are notorious for being unreliable in various ways, and optimizing for them is the cause of many sub-optimal decisions and Prisoner's Dilemmas in many industries.

> Customers like a product because it removes their problems. QED.

Your first point was "actually the world is really complicated" (strong agree) and your second was "actually the world is super simple" (strong disagree). Customers like products for wild reasons, unknown even to themselves. I've heard some slot-machine addicts get irritated if they win the jackpot, because it breaks them out of their state of flow. You could argue "well clearly that lack of flow was the problem slot machines are solving" but then you're just using circular reasoning: if a customer likes a product, it must be solving the problem for them that they previously did not have that product. I don't think you could have looked at the slot-machine addict a year before they started gambling and deduced that they had some sort of problem that slot machines needed to fill. There's often no clear link from a-priori diagnosable "problems" that people have to the products that they buy. In the universe populated by logical econs that do act this way, store shelves and advertising look very different.

I have no envy; I'm a software engineer as well. I just don't seem to struggle, as others claim to, in measuring productivity. I find it fairly straightforward to see that some people are more productive than others, at least in my workplace where we can hold most things constant (e.g. meeting count, managerial ability, etc, etc). Yes, there may be external factors, and that's unfortunate. Yes, it's not fair that some people are more productive than others despite putting in half the effort. But I'm not going to stick my head in the sand and pretend it's invisible.

Because you are measuring at a very broad and basic level.

Steve is more productive than Susan.

Great. How much more productive? Can you turn it into a number?

Can you still do it consistently when Steve and Susan are in different teams in different parts of the organisation trying to achieve different goals?

I've done DB upgrades that took 10 minutes and I've done DB upgrades that took 3-4 months. What changed was not my productivity but the nature of the problem. Yet from the outside they were both just DB upgrades.

If Susan had done the DB upgrade in 12 weeks could we confidently claim that Steve could have done it in 11 weeks? Steve hasn't even done a DB upgrade since he joined the company. Perhaps Steve could have done it in 10 minutes?

If Steve and Susan are in different part of organization, the answer is "cannot compare". If they are doing different job, the answer is the same.

But every once in a while there is a scenarios when you can compare people easily.

There has a weekly rotation to be an support person for other team. During his week, John always answers questions quickly and to the team's satisfaction. Meanwhile James struggles to answer them and cannot troubleshoot product his team is writing. This has been going on for multiple months and hundreds of questions for each, so it's not "bad week" unlucky or fluke. We now know who is better at answering questions about product.

John and James are doing DB migrations, they did many dozens of them. The migrations are assigned randomly. But John is usually finishing his migrations with no problems, while James often caused outages or missing data. A few times James took over two months to migrate, so the task was taken from him and given to John. John had to discard everything James did and migrate everything from scratch. Now there is a migration for a very important client and CEO is fed up with random assignment.. who is he going to choose?

Or maybe while John is technically adept, he's also a huge jerk and belittles people at standup, while James is the quintessential communicator with jr devs, etc.

Real life is messy. I've seen more people get replaced due to attitude or teamfit issues than specifically due to technical incompetence.

Your second scenario is unfortunately very likely, people are jerks, and if they are also high performers (or high bullshitters) then can get away with it.

Either way I fully agree one one should be firing/promoting people based on a single metric, even if that metric is very relevant to the job description. That doesn't mean that "there is no such thing", or that if you really need to get that DB migration done, you want to choose a "quintessential communicator with jr devs".

Agreed.

> If Steve and Susan are in different part of organization, the answer is "cannot compare". If they are doing different job, the answer is the same.

These are the situations where we would get the most value from the metrics though.

The team level already has an Engineering Manager or Tech Lead who can directly deal with team level problems.

This is moving goalposts. OP's argument was "There's No Such Thing as Software Productivity", not "You Can't Convert Software Productivity into a Floating Point Number With 3 Decimals of Accuracy."

Would that be a fairer assertion?

Then I shall move the goalposts. Can you address my shifted goalpost?

I personally did not interpret the author as literally meaning there is no such thing as software productivity but I agree the way he wrote it was confusing and could be interpreted that way.

Even in his toy example he clearly stated Peter did a better job than Frank.

I think that's the source of the problem: it's impossible to measure the "work" required to solve most software problems. If you tell me to carry a stone up a hill, I can put that in a formula and know exactly how much work I'll have to do. But if you give me a ticket to do a DB upgrade, I can, at best, make an educated guess.

So by the time I close the ticket, how much work have I done, and how do I know whether the time I've spent it proportional to the work I've done?

What are your eyes measuring? Are you being fooled?

For other teams yeah, it's harder, or maybe even impossible if the teams are isolated.

I'm not sure how true this really is. We all like to think that we're good at estimating other people's ability, particularly if we work closely with them, but how can you validate that impression?

Something that happened to me at one point in my career was that I joined a team that was half new hires with little experience, and half long-term employees with a lot of experience. Because the long-term employees were typically extremely busy, I ended up answering a lot of questions from the entry-level devs. So I spent time documenting the system, doing trainings, doing 1-on-1 code reviews, and so on.

During the performance review, feedback from the entry-level devs was that I was highly productive, and feedback from the long-term employees was that I was a slacker.

I stand by what I said, whether people like it or not.

Including the part about going off vibes. Please explain how this "you can just kind of tell" mentality is not literally just going off of vibes?

(And I wouldn't choose a cofounder only based on productivity, anyways. I'm not even sure that would be the main criteria.)

I should clarify that I actually agree with you if what you're saying is that the best measure of productivity we have is literally just going off your gut, but my argument is that this gut feeling is terrible. This is in large part because humans are biased, our gut feelings are swayed by things that simply shouldn't factor in. We tend to have more positive opinions of people that we think are like us and we sometimes wind up having negative opinions of someone based on stupid things like disagreeing with them on something that is ultimately irrelevant to whether or not they are productive.

And to me, the ultimate nail in the coffin is really in the question of what really constitutes productivity. Productivity is supposed to measure the efficiency of producing something, which already has pitfalls in and of itself when dealing with things that do have discrete, measurable indicators of progress, but programming doesn't, it's not even always obvious if progress is forward or backward sometimes. What's most important, performance optimizations, disaster planning, features, robustness, minimizing resource utilization? The best answer you can generally say is "It depends," and moreover, everyone will have a different set of competencies they're best at, so a person doesn't really have some single "productivity value" you could summarize them with. Yet, all of those things are pretty important for any serious technology organization, so you would want people with a variety of different affinities.

At that point, it starts to beg the question of whether or not attempting to directly measure software productivity is a worthwhile endeavor. I'd argue not.

I do believe that you can use basic tests to determine whether someone has more technical competence than some random guy off the street, but it only works up to a point. If you try to test deeper and deeper knowledge, you might create a mirage where someone who isn't very competent appears competent because you just happen to hit on strong spots on their very sparse experience. (My imposter syndrome reasons that this is why I passed the Google interview so easily some years ago.)

But for example, interviews don't even really bother trying to determine any direct proxies for productivity. Usually, they just stick to trying to determine technical competence, communication, ability to work on a team, and evaluate their history of technical accomplishments. A list of accomplishments is evidence of productivity, to a degree, but not having a long list is not evidence of a lack of productivity, and neither will tell you what will actually happen when you hire the person. References will at least give you someone else's gut feelings (or lies) regarding someone's productivity, but any reasonably competent person is going to have people who can vouch for them even if their productivity is actually not very impressive. It's not like there's some huge punishment for embellishing someone when you're being interviewed as a reference for them.

In the context of hiring people, gut feelings are probably the best thing we have, but they're subject to horrendous bias. Even if you are highly enlightened and can recognize your own biases with a great degree of humility, this is not generally the case for most people. Because of that, Google's interviews have a lot of layers of abstraction designed to eliminate bias from the process, but then again, they also wound up doing a study where they hired people who were ultimately turned down for the job and found that those people had around the same chance at succeeding at Google as the people who were hired. (Can't find the source for this because Google Search is useless nowadays. Maybe their hiring process is to blame.) And yet, there's no doubt that even with this in mind bias will still impact the interviews, because the interviewer does ultimately have to transcribe the interview and they can choose to omit or paraphrase things in a way that makes it look worse to the committee overseeing things; likewise, you can "correct" what the person is saying if you felt it was "close enough", or omit entire segments that looked weaker. Sure, you're not supposed to, but I would bet you 10:1 that even people not intending to be biased wind up doing this. Maybe they're second-guessing themselves when they do it: was it my fault they didn't answer better? Were they saying it right all along and I just wasn't understanding?

I was actually involved in a lot of interviewing and hiring especially early on in my career. I still believe gut feeling was the best instinct I had, but there was a time when I didn't agree with a hire and was proven horrifically wrong very soon after. Granted, that mostly comes down to how you evaluate someone's technical competence, not necessarily productivity, but I think the point stands either way.

> there was a time when I didn't agree with a hire and was proven horrifically wrong very soon after

Based on what? Did they turn out to be unproductive, but in a good way? What was that way?

Software developers should look at anyone claiming they can measure software productivity as a snake oil salesman.

We have seen hundreds of attempts over the years and they have all "failed". More accurately they all have large error bars and biases.

Researchers can and should continue looking into how to measure software development productivity. It is likely over the next few decades we will start to understand how to measure it appropriately.

The text gives an example to the core problem, and to argue differently requires thinking around it.

In practice. I’ve seen many attempts at measuring productivity, but once you dig into them, you see they are just abstraction mechanisms above something that is similar to lines of code.

I have yet to see an idea that sidesteps the core issue described in this post. Also, it applies to many types of work, and software is not unique in any way.

All other measures are a proxy for happy customers.

Actually, happy customers is also a proxy (the real measure is profits) but measuring profits directly (in the short term) can lead to decisions that have adverse long term effects. It's too easy to increase profits in the short term by avoiding long-term expenses.

So, if you're in the business of software, the goal is happy customers. (And I use the word Customers carefully here. Not just Users who pay nothing, but Customers who spend money.)

In a business context, it's really the only thing that matters. But, of course, it can be hard to measure (are they Happy?) and relies on multiple disciplines. Production (coding), Marketing, Sales, Support, Documentation, Training- all need to be working well to make it work.

Ultimately if the big picture doesn't lead to Happy Customers (again, I stress, in a businesses context) then no-one is "being productive."

So the business is chasing profits. In the short term that means customers paying money - any will do (happy or unhappy).

But in the long term, happy is the key. Happy customers are the single biggest marketing tool you have. Happy customers promote and recommend you. Unhappy customers do the opposite (and are more effective at doing so.)

So, if the metric stops at Customers then you are greatly missing the long-term value. Since a good business is planning for the long term, not just right now, Happy customers I the correct metric.

Remember, you ultimately get what you measure (no more).

And yet, there's some irrational counter examples; there's video games that has huge detractors while having huge financial success. Negative reviews on Steam, "4000 hours played". The metrics say they aren't happy with the product... but they still play it, talk about it, may have pulled in friends to play it, and spend money on it.

People can be unhappy about a product but still pay and promote it, counter-intuitively. Of course, the unhappiness is what they say, their behaviour says otherwise so for the sake of the metrics they would be considered happy I suppose?

As the other commenter pointed out, happy customers means nothing if they aren’t actively paying you

Even if you could perfectly measure customer happiness (very hard, as you note) - it's relatively easy to make customers happy by giving them more value than what they pay for. Sure, that may cost your business more money than what it makes with said customers, but hey, who cares, "profit" was not the metric...

(and as you note, if you make "profit" the metric, that has its own set of challenges - e.g. the optimization towards short-term profit in detriment of the long-term sanity, which is what we observe in a lot of corporations).

Yes and no in this case. Yes, you can naked customers happier with more value, more overhead (ie more support staff and do on.)

Yes, in the short term this might reduce profit. If you go too far down this road you might go bankrupt. No measure works if you dont use the "can we afford it" metric.

But nothing turbo-charges profits (in the short, and more importantly, long term) than happy customers. Ultimately they pay more, theny pay more often, they encourage others to pay.

If you optimize for happy customers, and stay solvent, you have the foundation for a solid long-term business.

I will add that starting with Happy Users (who get stuff for free) and turning them into Happy Customers later is really really hard. Simply giving the thing away (or charging so little it amounts to the same thing) is not what I'm suggesting. You can start with a lower price, yes, but regular price hikes are part of yhe process until uou find your natural price level.

Anyway there's this adage that once a metric (like productivity) becomes a target it ceases to be a useful metric. But this doesn't seem to apply for value / revenue much, so I suppose it's good to keep an eye on this vague productivity metric.

A potential riske seems to be feedback systems where job satisfaction is determined by high or low pay.

This is something that one of the orgs I worked for eventually realized. The people f'ing up, and then fixing their mistakes, were the ones getting promotions/bonuses/raises, because they were the ones interacting with all the execs.

Measuring developer productivity should, in my opinion, have one dimension for speed, one for quality, and one for user impact. LOC can be fine as a measurement for speed, you just don’t want to look at it in isolation. You would want to also measure, for example, escape rate and usage for the features the developer worked on, and be willing to change or refine these if circumstances require it.

You also need to look for different profiles based on the developer’s level of seniority. A senior dev probably shouldn’t be writing as much code as a contributor, but their user impact should be high, and escape rate low. Analyzing differences between teams is important, as well. A team that has a lot of escapes or little user impact probably has issues that need management attention, and may not have anything at all to do with individual developer productivity or ability.

In brief, the numbers are there to help you make better management decisions, not to relieve you of having to make them.

Wherein the work isn't repeating existing work no such measure should be expected to exist.

Any creative work is going to suffer from the same problem.

If you asked a brick layer to develop a better workflow for his fellows to work more effectively in a particular sitiation it would be the same. Are you going to measure words per minute?

Also imagine every other week as the bricklayer had half a wall built a PM comes out and says, "tear this all down and move it 3 feet to the north", and then the week after, "now move it east 1 foot, and why is it taking you so long?"

https://doi.ieeecomputersociety.org/10.1109/32.44364

So you can measure productivity with a reasonable level of accuracy and consistency, but then what? For most organizations it's not actionable so calculating that metric ends up as another form of waste.

Well enough for what? And how do you measure quality?

Developer productivity exists and I am totally comfortable describing Alice as being more productive than Bob. The fact that there isn’t a good, generic way to distill this into particular values does not mean that the phenomenon doesn’t exist.

We’ve all worked with Alice and Bobs. Claiming there’s no such thing as developer productivity is denying what we’ve all experienced. We aren’t special snowflakes whose work is beyond the ken of mortal men. Our work has value and sometimes we produce a lot of value in a particular time period and sometimes we product a little. This is productivity.

Calling this phenomenon a toy example is a bit out of touch. I've seen this every single day of my 25 year career. Value is produced by solving problems, not writing code. The solution with fewer lines of code, fewer new abstractions, lower complexity (yes, complexity IS objective and quantifiable) is invariably the best solution. Solutions that involve no code at all are the gold standard.

Adding/subtracting the right code produces value. But adding the wrong code decreases value. The logic of "productivity as code" contains a fatal flaw - it ignores this and assumes all code is right. Reality: everything hinges on the right vs wrong distinction which is inherently subjective. Code is too often a net liability - you must account for the risk!

By contrast, the "productivity as solving problems" approach is entirely objective. You can observe the problem, test hypotheses about the cause, and measure the situation after the intervention. A well stated problem has no ambiguity.

I don't see any support whatsover for the "productivity as code" idea. It's empirically false and lacks logical consistency.

Spoiler: they can't do that either. How do you measure the productivity of a bridge engineer designing a new suspension bridge? Number of struts placed on the blueprint? Even with more manual labor, it's very hard. Who's the most productive out of these three:

1. A carpenter who builds a house from a blueprint in 6 weeks. It collapses after 3 years.

2. A carpenter who builds a house from the same blueprint in 12 weeks. It stays up, but has visible defects that affect its value.

3. A carpenter who builds a house from the same blueprint in 20 weeks. It stays up for over a century and eventually becomes a historical landmark due to its lasting beauty and careful construction.

Which one was more productive?