> The experienced engineer also gets an LLM though

Say experienced engineer Edna was working on a project and the new grad Nelson replaced her because he was cheaper and about as good if both have LLMs, i.e.,

Edna + LLM ≈ Nelson + LLM

even though

Edna >> Nelson.

So Edna needs to find a project to work on where LLMs don't equalize her arbitrage, i.e.,

Edna + LLM >> Nelson + LLM

and Nelson needs one where

Nelson + LLM >> LLM.

My contention is "how do you determine if someone was 'as good'?" Very likely the people in the position to make a decision will not be qualified enough to even make that decision. IF (and really only if) such a determination is possible, then sure, I agree, find harder problems etc, but I assert that people will not make that determination (which is close to impossible to make) well, if at all because that wasn't the point in the first place.

Project succeeds = good.

Project fails = bad.

If a bunch of projects given to NG + LLM fail, the company will say at some point "this is no good, let's not do that anymore". If they mostly succeed, they'll do it more.

From a middle manager's perspective, where they're funding 10-20 projects, this kind of measurement is feasible and carries at least some signal. So I don't agree it's difficult to measure.