News for hackers. Stuff that matters.

A comment on A guide to open-source LLM inference and performance.

Re:A guide to open-source LLM inference and performance by alanaan on Monday November 20, 2023 @10:11PM (#38358571)
great post. could you apply this same framework to optimize training as well?
[ Parent | on HN ]
Re:A guide to open-source LLM inference and performance by varunshenoy on Tuesday November 21, 2023 @11:29AM (#38365832)
Slightly different set of trade-offs, but similar mental model. You always use large batch sizes (compute bound) and the bottleneck usually ends up communication between GPUs/nodes.
[ Parent | on HN ]