Refact LLM: New 1.6B code model reaches 32% HumanEval and is SOTA for the size

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by ttul on Monday September 04, 2023 @02:23PM (#37383342)

There are many MoE architectures and I suppose we don’t know for sure which OpenAI is using. The “selection” of the right mix of models is something that a network learns and it’s not a complex process. Certainly no more complex than training an LLM.

[ Parent | on HN ]

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by axpy906 on Monday September 04, 2023 @03:57PM (#37384336)

When I wrote “backend” was a poor choice of a word. “Meta-model” is probably a better choice of wording.

I hope it did not detract too much from the point of focusing on subtasks and modalities for FOSS as GPT 4 was built on a $163 million budget.

Finally, good point. We’ve got no idea of what OpenAI’s MoE approach is and how it works. I went back to Metas 2022 NLLB-200 system paper and they didn’t even publish the exact details of the router (gate).

[ Parent | on HN ]

Re:Refact LLM: New 1.6B code model reaches 32% HumanEval and... by ttul on Tuesday September 05, 2023 @01:47PM (#37395265)

Yeah, good point on the importance of FOSS focusing on subtasks... because FOSS isn't going to be spending $150M+ training a model any time soon without something like government backing.

[ Parent | on HN ]