News for hackers. Stuff that matters.

A comment on Refact LLM: New 1.6B code model reaches 32% HumanEval and is SOTA for the size. Parent comment

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by palmer_fox on Monday September 04, 2023 @03:14PM (#37383895)
Perhaps the wrong thread to ask this question... Is it not possible to load a model on something like an NVMe M.2 drive instead of RAM? It's slower of course, but only 5-10x if I understand correctly.
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by kirill5pol on Monday September 04, 2023 @03:42PM (#37384171)
Yes but they’re slow enough on normal hardware for that 5-10x to be painful…
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by mirekrusin on Tuesday September 05, 2023 @02:55AM (#37388697)
Can you RAID them?
[ Parent | on HN ]
Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by brucethemoose2 on Tuesday September 05, 2023 @09:48AM (#37391588)
Technically yes?

But its way beyond the point where its going to help LLMs. CPU RAM is already "too slow" in machines big enough for multiple NVMe SSDs.

[ Parent | on HN ]