Ultimately the performance will be assessed via the income statement and cash flows of customers of the model producers.
Frankly in the window pre-IPO it’s in the best interests of OAI et al to show a line going to the top-right in relation to tokens, in their prospectus. What does that mean?
Strategic manipulation.
As best as I can tell, that's the thinking. It's one number, it's very easy to find and manage, and there is a belief that it directly measures productivity.
I disagree that it does; seems to me the throughput of useful features is a better measure, but I'm not in the drivers seat on this one
The personal use case stuff is messy and subjective.
Cost savings is primarily a function of headcount here. Which is also easy to measure, and so if we take my thesis that easy to measure stuff is prioritized...
This includes encouraging people to set up elaborate multi model set ups (e.g. "gas town") for coding that do not meaningfully improve productivity but which certainly do cause token usage to explode.
It also includes encouraging execs to use token consumption as a proxy for productivity - almost akin to SLOC.
AI has a halo right now and the managerial class seem to be willing to forgive almost any failure because the promise is so enticing. We're at peak expectations right now. They will soon start to be less forgiving when the warts which are intrinsic to LLMs remain unsolved.