Except that this isn't the argument I'm making, and I don't think most of the people who are pointing it out are claiming that inference being sold at high gross margins means that you don't have to consider the massive investments that have been made.
The more common narrative is that these companies are losing massive amounts of money and will never be able to make a profit. And that's where looking at the investment in training versus the economics of inference is useful because, once again, inference actually does have high gross margins.
> R&D makes sense if we consider that research and development eventually gets to a model that doesn't need to be trained anymore, and can be served at relatively low cost indefinitely.
This just isn't true. If revenue grows at a sufficient enough rate versus training costs, and/or training costs decrease substantially enough, the numbers could work out over time.
People who are claiming that it's mathematically impossible are going on vibes. And just to be clear: I'm not claiming that it working out over time is guaranteed, or the likely outcome. But it is possible and the mechanics by which it would work out (if that comes to pass) is pretty simple.
> I agree that this is speculation. But it is based on actual sweetheart deals taking place (such as SpaceX not charging for compute during a period in Q2) and on Anthropic being very unclear if their compute cost is based on market rates or subsidized by providers who are also investors in the company.
Regarding SpaceX: my understanding is that as it was ramping up its capacity, which was not optimized and capable of being delivered in full at the time, it provided concessions, including discounts and deferred billing, to large customers. What SpaceX got from this in return was a massive backlog (over $14 billion worth) of non-cancellable contracts.
The concerns about Anthropic investors are legitimate, but again, you can only speculate. If your entire thesis ignores plausible scenarios that aren't so negative (like the possibility that compute is being sold at or near market rates), you have to be honest: the thesis is biased to support the argument you want to make.
If I was going to invest in any of these companies, I would want to know more about the terms of these deals but I don't think anyone who hasn't actually seen the agreements should make confident arguments based on what they think must be in them.
As a side note: if you really want to be exact, we'd need to know how compute purchased at below cost affects the income statement. If the below-market pricing was for compute related to training, it would not be the same on the income statement as compute related to inference. Whereas below-market pricing for inference compute would hit COGS and improve gross margins.
> You may have a major breakthrough that halves the compute use for a task. It is meaningless if the hardware is now 3 times more expensive.
What matters is the price per unit of compute. "The hardware is more expensive" only threatens viability if the price per unit of compute is rising, not just the price per chip.
You're throwing around random numbers to make a point but there is real data on this subject. For example, Epoch AI published a report a couple of months ago that found the cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023 even as hardware and memory prices rose by tens of percent. On one of its science benchmarks the cost per question at a fixed score fell 725x in less than 18 months.
It did also find that large training runs have grown by 2-3x per year, which is the crux of everything, once again: training costs versus inference costs.
> People who are claiming that it's mathematically impossible are going on vibes.
You are doing the same thing you claim I am, just on the opposite side of the road.
In truth this is sentiment based on the current narratives.
To be clear, I don't think it is theoretically impossible to be able to recoup the cost of training by selling inference, but I do think it is impossible with the current rate new models have to be trained and with current cost they charge customers. I believe that they will have to increase prices pretty dramatically for this to make some sense.
> As a side note: if you really want to be exact, we'd need to know how compute purchased at below cost affects the income statement. If the below-market pricing was for compute related to training, it would not be the same on the income statement as compute related to inference. Whereas below-market pricing for inference compute would hit COGS and improve gross margins.
My whole point is that this is unclear. Anthropic's S-1 mentioned how much that they paid for compute, but the rate they pay is totally opaque.
If they pay near the market price this is a nothingburger. If they pay, say, half the market price, now we can see their already eye-watering training costs increase significantly and their high profit margins to suddenly not look as seductive anymore.
Again, my annoyance is how prople pretend that Anthropic has a clear path to become profitable because of ther incredible revenue growth and profit margins (even if you are not doing so, this is the pervasive narrative in forums like HN). I sustain that this is far from settled.
Except that I'm not on the opposite side of the road. I've simply pointed out that the mechanics by which Anthropic could be viable long-term are pretty simple; not that this is what is going to happen.
Frankly, I don't think anyone can predict what will happen. This is a fast-moving, complicated stew of technology, finance and politics.
> My whole point is that this is unclear. Anthropic's S-1 mentioned how much that they paid for compute, but the rate they pay is totally opaque.
> If they pay near the market price this is a nothingburger. If they pay, say, half the market price, now we can see their already eye-watering training costs increase significantly and their high profit margins to suddenly not look as seductive anymore.
Pricing in commercial agreements is typically considered competitively sensitive and can be redacted from contracts filed with the SEC but only if it isn't material. Anthropic still has to disclose known trends that are likely to affect its results so if below-market compute were inflating its margins you'd expect to see that in the S-1.
The more detailed related-party disclosures probably aren't required yet because Amazon and Google mostly hold non-voting preferred stock and convertible notes. The S-1 says they'll each hold more than 5% after the IPO conversions. At that point, their transactions with Anthropic fall under the related-party rules so we'll learn a lot more after the IPO if there are any "issues" there.
> The more detailed related-party disclosures probably aren't required yet because Amazon and Google mostly hold non-voting preferred stock and convertible notes. The S-1 says they'll each hold more than 5% after the IPO conversions. At that point, their transactions with Anthropic fall under the related-party rules so we'll learn a lot more after the IPO if there are any "issues" there.
Whoa, that I really didn't know. Something for everyone to keep their eyes on then, because this is actually very opaque, and may dramatically change what we know about their numbers. Or not.
The problem I have with this is that the claim that "inference has high gross margins" is paraded around as something that automatically makes all this make sense.
My concern here is something that you described with more nuance, the cost of training cannot just set aside as if it doesn't need to be recovered. R&D makes sense if we consider that research and development eventually gets to a model that doesn't need to be trained anymore, and can be served at relatively low cost indefinitely. Otherwise the constant need to train new models needs to be taken into account; you can only keep revenue by training new ones.
> You have absolutely no evidence of "sweetheart compute deals" so this is pure speculation.
I agree that this is speculation. But it is based on actual sweetheart deals taking place (such as SpaceX not charging for compute during a period in Q2) and on Anthropic being very unclear if their compute cost is based on market rates or subsidized by providers who are also investors in the company.
> Furthermore, there's a flaw in your logic. It's entirely possible for chips and memory to get more expensive while the cost per token served falls due to more efficient hardware and better inference software. What matters here is the cost per unit of output, not the price of a GPU.
I agree that my statement does not take into consideration hypothetical improvements in efficiency, but I dispute that the increased cost of hardware is not a menace to their viability.
You may have a major breakthrough that halves the compute use for a task. It is meaningless if the hardware is now 3 times more expensive.