Re:I think Anthropic and OpenAI have found product-market fit
by whatshisface on Wednesday May 27, 2026 @07:04PM
(#48301962)
That is an estimate of the relative cost of one training step, but you have to multiply it by the number of training steps, an unknown quantity.
Re:I think Anthropic and OpenAI have found product-market fit
by janalsncm on Wednesday May 27, 2026 @04:55PM
(#48300536)
So one way to think about it is roughly,
Training is inference + backwards pass (~2x inference cost) + activations (vram overhead) + optimizer (vram overhead) + gradients (vram overhead).
Re:I think Anthropic and OpenAI have found product-market fit
by thesz on Wednesday May 27, 2026 @05:29PM
(#48301007)
Multiply "inference + backwards pass (~2x inference cost) + activations (vram overhead)" by batch size (thousands) to get to the actual RAM and compute cost. Optimizer like ADAM adds only two or three model-sized overhead.
And last, but not least, you need only one hidden layer kept in RAM for inference, but you need all of them (61 for Deepseek models) kept in RAM for computing gradient for one sample.
Re:I think Anthropic and OpenAI have found product-market fit
by xyhopguy on Wednesday May 27, 2026 @07:30PM
(#48302209)
Microbatch size is a hyperparameter, it can be set to 1 and work just as effectively. With gradient accumulation it's equivalent even. Large batch sizes are used to increase parallelism, and sometimes to reduce variance in the loss signal (at the cost of increased bias).
Batch size is frequently limited by compute bottlenecks well before memory.
Re:I think Anthropic and OpenAI have found product-market fit
by galaxyLogic on Wednesday May 27, 2026 @07:14PM
(#48302069)
Does it matter what is the difference in size of needed inputs for inference vs. training?
Re:I think Anthropic and OpenAI have found product-market fit
by mcv on Thursday May 28, 2026 @04:18AM
(#48306159)
And of course you do all of this for every object in your training set, which is going to be larger than the total number of uses for any individual user.
Re:I think Anthropic and OpenAI have found product-market fit
by mike_hearn on Thursday May 28, 2026 @07:38AM
(#48307521)
It's all got much more complex than that in recent years. Training now involves large amounts of inference for RL rollouts and similar. You can't disentangle them computationally like that. "Inference" is just the word used to mean serving customer traffic now, and "training" means creating the model you serve.
Training is also done over batches, which increase memory requirements by several orders of magnitude. This is why training needs costly compute.
One of the ways out of this unfortunate situation is to use something like Stochastic Average Gradient Descent [1]. Examples there are mostly concerned with regularized logistic regression, which makes problem more or less convex. Neural networks are inherently non-convex. Still, maybe some ideas from there can be utilized in the context of neural networks, like use of estimated Lipshitz constant to derive curvature and appropriate learning step.