If a new grad + LLM can do task X about as well as an experienced engineer can, then the experienced engineer will have to find other harder problems to solve. Personally I think labor efficiency is good; why waste your labor on easy/solved problems?
The experienced engineer also gets an LLM though
Say experienced engineer Edna was working on a project and the new grad Nelson replaced her because he was cheaper and about as good if both have LLMs, i.e.,
Edna + LLM ≈ Nelson + LLM
even though
Edna >> Nelson.
So Edna needs to find a project to work on where LLMs don't equalize her arbitrage, i.e.,
Edna + LLM >> Nelson + LLM
and Nelson needs one where
Nelson + LLM >> LLM.
Project fails = bad.
If a bunch of projects given to NG + LLM fail, the company will say at some point "this is no good, let's not do that anymore". If they mostly succeed, they'll do it more.
From a middle manager's perspective, where they're funding 10-20 projects, this kind of measurement is feasible and carries at least some signal. So I don't agree it's difficult to measure.
I actually don't relate to this "bosses don't know what software engineering is" thing at all. All my bosses are older-school engineers-turned-management who are more skeptical of this than the young'uns.
Which is why bosses are excited but engineers are not, because subpar LLMs will be used to justify firing actual engineers and replace them with fresh college grads tasked to work with a code bot, because the bosses fundamentally think that is enough.