BigCode Project Releases StarCoder: A 15B Code LLM

Re:BigCode Project Releases StarCoder: A 15B Code LLM by enum on Thursday May 04, 2023 @10:49PM (#35824700)

There is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLMs.

But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.

[ Parent | on HN ]

Re:BigCode Project Releases StarCoder: A 15B Code LLM by EvgeniyZh on Friday May 05, 2023 @01:32AM (#35825544)

You are correct that there is no evaluation of level of support, and it is hard to get evaluation set on 86 languages. On the other hand, we can try to extrapolate from what we have and try to guess in which languages we'd have reasonable performance. Note that out of 19 languages it was evaluated on only 17 were "officially" in the training set (language detection is not perfect and there may be some data for languages not included in training set) and it work reasonably well on the remaining two (Swift and D).

[ Parent | on HN ]