News for hackers. Stuff that matters.

A comment on Jamba: Production-grade Mamba-based AI model. Parent comment

Re:Jamba: Production-grade Mamba-based AI model by nl on Friday March 29, 2024 @03:53AM (#39861600)
This is wrong. You need big memory during inference too.

The difference there is you can use tricks like quantisation and offloading to CPU to reduce it somewhat at the cost of accuracy and/or speed.

[ Parent | on HN ]
Re:Jamba: Production-grade Mamba-based AI model by brrrrrm on Friday March 29, 2024 @09:45AM (#39864049)
Training is 3x the memory used by inference, and usually run at a much larger batch size
[ Parent | on HN ]
Re:Jamba: Production-grade Mamba-based AI model by pama on Friday March 29, 2024 @09:56PM (#39871125)
Not sure what you mean by wrong. I have never encountered a case yet when training an LLM (no matter what architecture) would require limited memory and was pointing out that the typical memory requirements for training are much higher yet than the typical requirements for inference.
[ Parent | on HN ]