Small offline large language model – TinyChatEngine from MIT

Re:Small offline large language model – TinyChatEngine from MIT by physicsgraph on Monday December 18, 2023 @07:08AM (#38681448)

Thanks for the suggestion. I'm new to running LLMs so I'll take a look at your suggestion [0]. My ~10 year old MacBook Air has 4GB of RAM, so I'm primarily interested in smaller LLMs.

[0] https://github.com/ggerganov/llama.cpp

[ Parent | on HN ]

Re:Small offline large language model – TinyChatEngine from MIT by akx on Monday December 18, 2023 @07:12AM (#38681484)

You don't necessarily need to fit the model all in memory – llama.cpp supports mmaping the model directly from disk in some cases. Naturally inference speed will be affected.

[ Parent | on HN ]