Re:The Asus Ascent GX10 a Nvidia GB10 Mini PC with 128GB of ...
by buildbot on Thursday March 20, 2025 @05:33PM
(#43429246)
AI inference is actually typically bandwidth limited compared to training, which can re-use the weights for all tokens <sequence length> * <batch size>. Inference, specifically decoding, requires you to read all of the weights for each new token, so the flops per byte are much lower during inference!
[ Parent
| on HN ]
There are no comments on this comment yet.