Re:Jim Keller criticizes Nvidia's CUDA, x86
by mnky9800n on Friday February 23, 2024 @11:53AM
(#39483037)
What does it mean by "inference" in this context?
[ Parent
| on HN ]
Re:Jim Keller criticizes Nvidia's CUDA, x86
by hangsi on Friday February 23, 2024 @12:35PM
(#39483625)
Neural networks have two different compute costs: training and inference.
These are roughly analogous to compile time vs runtime for compiled programming languages.
Training is in general a more intensive task. However, in an ideal scenario training is run once and inference is run millions of times, so the lifetime cost of inference is bigger - this is why it might make sense to optimize for intense.
Re:Jim Keller criticizes Nvidia's CUDA, x86
by imtringued on Friday February 23, 2024 @05:57PM
(#39487221)
Inference consists of a single forward pass, which is easy to compute. Meanwhile training has both a forward pass and backward pass (back propagation). The minimum required precision for training is higher since you want to be able to have both fast and slow rates of change. Bfloat16 is preferred for training, while int8 tends to be good enough for inference.
Re:Jim Keller criticizes Nvidia's CUDA, x86
by syntaxing on Friday February 23, 2024 @12:31PM
(#39483576)
You’re trying to get an output from giving an input to a bunch of matrices (tensors if you will). You do not touch anything in the matrices themselves which happens during training. Essentially, they’re great from deploying models but not making one.