News for hackers. Stuff that matters.

A comment on BigCode Project Releases StarCoder: A 15B Code LLM. Parent comment

Re:BigCode Project Releases StarCoder: A 15B Code LLM by espadrine on Thursday May 04, 2023 @01:53PM (#35819465)
A similar model from Replit was released this week: https://huggingface.co/replit/replit-code-v1-3b

The performance of StarCoder seems superior on HumanEval pass@1:

• replit-code-v1-3b: 21.9%

• replit-finetuned-v1-3b: 30.5%

• StarCoder: 33.6%

• StarCoder prompted: 40.8%

[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by enum on Thursday May 04, 2023 @02:12PM (#35819750)
Also note that StarCoder itself is open (but not replit-finetuned).
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by kartayyar on Thursday May 04, 2023 @02:58PM (#35820410)
The quality delta is so huge that it's in a different league.

Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.

[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by sillysaurusx on Thursday May 04, 2023 @05:22PM (#35822259)
If so, it’s a shame this post is getting comparatively little attention. Hopefully the quality will be enough for it to make a big splash somewhere.
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by sauwan on Thursday May 04, 2023 @03:14PM (#35820663)
I'm assuming higher is better? How does this compare to GPT-3.5/4?
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by qwerty3344 on Thursday May 04, 2023 @03:25PM (#35820822)
https://newatlas.com/technology/gpt-4-reflexion/
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by qwerty3344 on Thursday May 04, 2023 @03:25PM (#35820821)
I think it's still significantly behind GPT 3.5/4, both of which can get 67% on HumanEval, and 88% with Reflexion
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by enum on Thursday May 04, 2023 @04:53PM (#35821907)
Keep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by manojlds on Thursday May 04, 2023 @05:14PM (#35822170)
Aren't GPT-3 etc base LM and ChatGPT the instruction tuned? Or am I wrong?
[ Parent | on HN ]
Re:BigCode Project Releases StarCoder: A 15B Code LLM by dpf on Friday May 05, 2023 @06:52AM (#35827595)
code-davinci-002 is a base LM, and the other 3.5 models (text-davinci-{002,003}, gpt-3.5-turbo, and ChatGPT) use instruction tuning and/or RLHF. Source: https://platform.openai.com/docs/model-index-for-researchers
[ Parent | on HN ]