Refact LLM: New 1.6B code model reaches 32% HumanEval and is SOTA for the size

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by nacs on Monday September 04, 2023 @05:16PM (#37385152)

Yes, 7B is perfectly usable on low-end hardware if you're using it for instruction tuning/chat.

But for code completion in an IDE where it has to react as you type, every 100 millisecond delay in response time is noticable.

Even with a 24GB GPU, a 7B model doesn't feel snappy enough for code-completion in an IDE.

[ Parent | on HN ]

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by brucethemoose2 on Monday September 04, 2023 @06:32PM (#37385764)

This can be addressed with token streaming and input caching.

Would that be enough? shrug

[ Parent | on HN ]

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by evolve7942 on Monday September 04, 2023 @11:37PM (#37387569)

GPU RAM quantity isn’t typically correlated to inference rate. Precision/quantization levels do affect model size, which will affect inference rate. However, I would expect a smaller model to be faster (less RAM).

[ Parent | on HN ]

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by brucethemoose2 on Tuesday September 05, 2023 @10:01AM (#37391741)

Llama (and many other llms, I presume) are so memory bandwidth bound that model size is a decent indicator of inference rate.

The smaller the model, the less has to be read from ram for every single token.

Batching mixes up this calculus a bit.

[ Parent | on HN ]