News for hackers. Stuff that matters.

A comment on Small offline large language model – TinyChatEngine from MIT. Parent comment

Re:Small offline large language model – TinyChatEngine from MIT by TotalCrackpot on Monday December 18, 2023 @10:23AM (#38683665)
Btw, shouldn't it in theory be possible to run the Mixtral MoE loading next submodel sequentially and store outputs and then do the rest of the algorithm to make it easier to run on machines that cannot fit whole model in the memory?
[ Parent | on HN ]
Re:Small offline large language model – TinyChatEngine from MIT by wfhpw on Monday December 18, 2023 @11:06AM (#38684334)
Yes but loading weights into memory takes time
[ Parent | on HN ]
Re:Small offline large language model – TinyChatEngine from MIT by TotalCrackpot on Monday December 18, 2023 @11:11AM (#38684405)
Yeah I imagine sequential inference would be slower. How long do you have to wait to load these weights on a personal PC? I have not tried using those systems so far.
[ Parent | on HN ]