I dunno: this is pure anecdata but software engineers I know well - including me - all say they’re delivering multiple times faster, so it’s definitely good for something.

I don’t know about other tasks, but it’s pretty rad for coding, testing, DevOps, and config.

This research is specifically NOT about tech companies. It is about the rest of the fortune 500.

If you are in software or IT at a non-tech company that is in the fortune 500, the impact of significantly better productivity may not be measurable in aggregate in the company's P&L.

In smaller companies I expect it could be more measurable because some of them really are using AI much more effectively across many functions. But we're finding it very hard to measure meaningful outcomes.

But actually 40% of the S&P 500 are tech companies, and they are saying only the mag 7 are showing clear revenue increase. Would would suggest that even in most tech we are not seeming measurable revenue growth, right?

One plausible explanations that has occurred to me is this: While programmers etc. can produce way more code with LLMs, the declines in quality and coherence of the work, and a widening gap with a coherent strategy and aim for value means the increased activity is mostly not valuable. I suspect this may just be an intensification of https://en.wikipedia.org/wiki/Productivity_paradox#Little_pr...

The mag 7 are getting direct or nearly direct revenue from AI inference. New money is pouring into that - it is not zero sum. Everyone else is getting internal productivity lifts that are matched by competition, and when they are deploying inference features they are usually ones their customers didn’t ask for and don’t value, OR that they expect everyone to have and are not a differentiator.

I think you are also correct that a lot of value is being wasted with projects that are developed impressively fast but that provide little value. It could be those are simply surplus personnel trying to add value or make people think they do.

> Everyone else is getting internal productivity lifts that are matched by competition

The main metric analyzed in the note is

> Revenue per worker

This can increase regardless of competition, but I guess would require either expansion into new domains or layoffs. So maybe we will see this raise as a lagging indicator if the LLMs are actually delivering the goods.

> when they are deploying inference features they are usually ones their customers didn’t ask for and don’t value, OR that they expect everyone to have and are not a differentiator

I am definitely seeing lots of this happening. Tons of energy going into side quests that aren't critical for anything. Meanwhile, strangely, the central features we actually need to ship, but that require deep redesigns and careful refactors in legacy code, are not moving quickly at all.

Increased productivity is caused by mass layoffs, not by AI adoption.

In case you are not being sarcastic, layoffs obviously don't "cause" productivity increase. [Edit: but they would help the main metric (Revenue per worker) analyzed in the note. so perhaps that is what we will see.]

Not sarcastic, it's how productivity is usually measured when you see any kind of stats. Not features, not lines of code. Revenue (or GDP if we talk about the country) per worker

AI EM here, same. The level of productivity has increased dramatically in the last 2 months. Reliable code also has increased due to AI.

This train isn't stopping.

What trips up a lot of SEs where I work is their techniques. Everyone who follows our process delivers faster and with higher reliability so far.

Same story here in my first hand experience + colleagues experience in corporate strategy, FP&A, Marketing and corporate development.

But is it churn or value? Most of the software I use has gotten visibility worse as you deliver multiple times faster.

I would pay extra for a company that just slowed down and moved deliberately.

People are not reliable narrators on their own productivity. They often mistake activity for productivity.

I’ll take that over “hmm, the average revenue per employee of these 493 massive companies is flat, so AI doesn’t add value”

How come? Revenue seems like a more reasonable metric in the context of companies than self reported 'productivity.'

For starters, because an average of 493 companies of all types and cultures tells you very little about whether AI is having an impact. For all we know, half these companies have implemented AI and are growing like crazy, and the half that haven't are losing revenue at a rapid clip, but it all averages out to zero.

I'm not sure I'm any faster, but I think the end result is better and I've attempted some big refactors and new architectures that I would never have tried without AI.

N=1 but I'm not sure it's made my bosses any richer. What we're shipping has better tests and documentation than it ever would have before AI.

Most, not all, are just delivering more bad stuff faster. I personally also think I deliver more good stuff with AI, but the products I use are just getting worse daily it feels like. Just the other day I noticed Slack had installed some disgusting menu bar icon for their AI bot, without asking, just to take a fresh example of this.

If only I wasn’t using all that productivity on fixing vulnerabilities

One weird trick that devs learn to deliver faster: simply just drop all guard rails and push directly to prod. We need to make sure our velocity report looks good after all.

A friend of mine works in fashion. Very briefly for background: there are runway models (who walk on live shows), and catalogue models (who work in studios presenting clothes against a neutral background in standard poses, more or less as living mannequins). The latter are being increasingly replaced by AI.

> all say they’re delivering multiple times faster

Are you measuring the right metric?

This feels like an on going lack-of-understanding in the industry.

In manufacturing, economies of scale and automation work because you're physically making more objects and objects inherently have value.

For software, making more software doesn't mean you can sell more units of software. The unit price of selling existing software is already 0. Economies of scale do not exist.

The OP says: net business output is not up as a result of AI.

So:

…if your team is doing more work and being more productive…

That just means the productivity of your team (or software teams in general) maybe doesn't have a meaningful business impact.

You should be worried if so. That means productivity = work force reduction (reduce fixed costs for the business).

Hold on, you’re claiming that software companies don’t have economies of scale? Is that right?

It sounds like you’re also saying that creating more software can’t result in more revenue?

We work in very different software industries.

My employer - one of the Fortune 500 - has been on the "everything AI" train. The fun part is that the same management which harps on measurable outcomes when asked about AI adoption have the same "AI boosts productivity".

They even brought in McKinsey who I believed, if nothing else, can spin up a good yarn. But even they couldn't create a metric based assessment other than "productivity gains".

Now I am not saying there is no productivity gains. Surely there is but I am always curious what metric people are using to measure these gains.

So if you are delivering multiple times faster I have to ask - How much time are you saving and how many employees worth of work you are doing now? 2-3-5 headcount worth of work? If yes, how do you calculate those numbers? And lastly would you put your money where you mouth is and if your team is lets say 10 person team and the company asks to reduce it down to say 2-4-5 people - would you take that bet confidently to continue with the same pace and produce quality work?

I've never heard any such claims but meanwhile in my actual job I'm seeing unbelievable productivity gains (in terms of actual cash spent before and after vs the same kind of work being done).

It's unclear how you turn that into more profits though, especially if all of your competitors are doing the same thing.

They are not wrong, they're lying. Very important difference. There are a lot of people who will say "I'm getting mega productivity boosts" who aren't getting financial returns from you believing them; those people are wrong.