News for hackers. Stuff that matters.
Re:The Asus Ascent GX10 a Nvidia GB10 Mini PC with 128GB of ... by buildbot on Thursday March 20, 2025 @05:33PM (#43429246)
AI inference is actually typically bandwidth limited compared to training, which can re-use the weights for all tokens <sequence length> * <batch size>. Inference, specifically decoding, requires you to read all of the weights for each new token, so the flops per byte are much lower during inference!

There are no comments on this comment yet.