News for hackers. Stuff that matters.

A comment on Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week. Parent comment

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by runnerup on Wednesday May 03, 2023 @01:54PM (#35805490)
They trained on https://huggingface.co/datasets/bigcode/the-stack-dedup which is a massive curated dataset accumulated from GitHub. Details are here: https://www.bigcode-project.org/docs/about/the-stack/

Many of the most-represented "languages" on GitHub are actually things like JSON, XML, HTML, CSV, text, markdown, YAML, and SVG.

More details from them here: https://blog.replit.com/llm-training

[ Parent | on HN ]

There are no comments on this comment yet.