Evidence? I’m sure someone will argue, but I think it’s generally accepted that inference can be done profitably at this point. The cost for equivalent capability is also plummeting.
Here you go: https://www.wsj.com/livecoverage/stock-market-today-dow-sp-5...
There's no such thing as "profitable inference". A company is either profitable or it isn't.
Let's for a second assume all the labs somehow manage to form a secret OPEC-style cartel that agrees to slow training to a halt, and nobody notices or investigates. This is already hard to imagine with the amount of scrutiny they're under and given that China views this as a military priority. But let's pretend they manage it. These firms also have lots of other costs:
• Staffing and comp! That's huge!
• User subsidies to allow flat rate plans
• Support (including abuse control and handling the escalations from their support bots)
• Marketing
• Legal fees and data licensing
• Corporate/enterprise sales, which is expensive as hell even though it's often worth it
• Debt servicing (!!)
• Generating returns for investors
Inferencing margins have to cover all of those, even if progress stops tomorrow and the RoI to investors has to be likewise very large, so margins can't be trivial. Yet what these firms have said about their margins is very ambiguous. As they're arriving at this statement by excluding major cost components like training, it's not clear what they think the cost of inferencing actually is. Are they excluding other things too like hw depreciation and upgrades? Are they excluding the cost of the corporate sales/support infrastructure around the inferencing?
But also, "profitable inference" IS a thing! "Gross margin" is important and meaningful, even if a company has other obligations that mean it's overall not profitable.
You mention "debt servicing", but OpenAI has no debt. All the money they have raised is equity not debt.
Have any of you tried re-producing an identical output, given an identical set of inputs? It simply doesn't happen. Its like a lottery.
This lack of reproducibility is a huge problem and limits how far the thing can go.
However, we can start by claiming that non-determinism is not necessarily a bad thing - non-greedy token sampling helps prevent certain degenerate/repetitive states and tends to produce overall higher quality responses [0]. I would also observe that part of the yin-yang of working with the agents is letting go of the idea that one is working with a "compiler" and thinking of it more as a promising but fallible collaborator.
With that out of the way, what leads to non-determinism? The classic explanation is the sampling strategy used to select the next token from the LLM. As mentioned above, there are incentives to use a non-zero temperature for this, which means that most LLM APIs are intentionally non-deterministic by default. And, even at temperature zero LLMs are not 100% deterministic [1]. But it's usually pretty close; I am running a local LLM as we speak with greedy sampling and the result is predictably the same each time.
Proprietary reasoning models are another layer of abstraction that may not even offer temperature as knob anymore[2]. I think Claude still offers it, but it doesn't guarantee 100% determinism at temperature 0 either. [3]
Finally, an agentic tool loop may encounter different results from run to run via tool calls -- it's pretty hard to force a truly reproducible environment from run to run.
So, yeah, at best you could get something that is "mostly" deterministic if you coded up your own coding agent that focused on using models that support temperature and always forced it to zero, while carefully ensuring that your environment has not changed from run to run. And this would, unfortunately, probably produce worse output than a non-deterministic model.
[0] https://arxiv.org/abs/2007.14966 [1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in... [2] https://learn.microsoft.com/en-us/azure/ai-foundry/openai/ho... [3] https://platform.claude.com/docs/en/about-claude/glossary
This world of extremes is annoying for people who have the ability to think more broadly and see a world where deterministic systems and non-deterministic systems can work together, where it makes sense.
And even say the latter strategy works - ads are driven by consumption. If you believe 100% openAI's vision of these tools replacing huge swaths of the workforce reasonably quickly, who will be left to consume? It's all nonsense, and the numbers are nonsense if you spend any real time considering it. The fact SoftBank is a major investor should be a dead giveaway.