Llama.cpp 30B runs with only 6GB of RAM now

Re:Llama.cpp 30B runs with only 6GB of RAM now by htrp on Friday March 31, 2023 @06:00PM (#35394244)

Just shows how inefficient some of the ML research code can be

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by robrenaud on Friday March 31, 2023 @06:57PM (#35394895)

Training tends to require a lot more precision and hence memory than inference. I bet many of the tricks here won't work well for training.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by actually_a_dog on Friday March 31, 2023 @07:07PM (#35394991)

As a former grad student, I can tell you, that's all research code, not just ML, or even "performance-oriented" research code.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by rvz on Friday March 31, 2023 @08:47PM (#35395797)

Exactly.

It also shows the number of impostors in this thread and inflated titles of self-proclaimed 'seniors' who can't optimize ML code to even be on the same league as Tunney (jart), and Gerganov (ggerganov).

Not even ChatGPT or Copilot could even submit a change or in-fact completely rewrite and optimize this code like they have done.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by visarga on Saturday April 01, 2023 @12:14AM (#35397055)

Remember this moment when you're about to criticise LLMs. People can act suboptimal too, even experts.

[ Parent | on HN ]

Re:Llama.cpp 30B runs with only 6GB of RAM now by alduin32 on Friday March 31, 2023 @10:23PM (#35396415)

For now we've just shown how measuring memory consumption can be tricky at times.

[ Parent | on HN ]