Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by waffletower on Wednesday May 03, 2023 @11:51AM (#35803856)

No Clojure. No Julia. No Haskell. No Racket. No Scheme. No Common Lisp. No OCaml. And, as much as I despise Microsoft, No C#. No F#. No Swift. No Objective-C. No Perl. No Datalog. A glaringly lacking choice of languages.

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by Dayshine on Wednesday May 03, 2023 @12:39PM (#35804478)

C# was available in the dataset they link, and is the most glaring ommission by global usage...

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by ebiester on Wednesday May 03, 2023 @12:07PM (#35804083)

I'm sure that has to do with the dataset available to them.

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by runnerup on Wednesday May 03, 2023 @12:49PM (#35804614)

Which is a deduplicated version of this: https://www.bigcode-project.org/docs/about/the-stack/

And probably, yes. While it contains 358 programming languages, obviously there's a long tail after the 20 most-represented languages. Some people might not expect without thinking about it for a bit that many of the most-represented "languages" are actually things like JSON, XML, HTML, CSV, text, markdown, YAML, SVG.

Also note that it won't be able to parse natural language nearly as well without additionally being trained on something like the LAION dataset, so this version will be more of an autocomplete like Copilot rather than something which can manifest high level business logic from whole cloth like ChatGPT.

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by ubertaco on Wednesday May 03, 2023 @01:24PM (#35805101)

I fed it some OCaml and it worked, though the example was trivial:

    type point = { x: int; y : int }
    let manhattan_distance (a: point) (b: point) : int =

which it completed to

    type point = { x: int; y : int }
    let manhattan_distance (a: point) (b: point) : int =
        abs (a.x - b.x) + abs (a.y - b.y)

...which is a valid and correct OCaml definition of this method:

https://try.ocamlpro.com/#code/type'point'='$4'x:'int;'y':'i...

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by mclide on Wednesday May 03, 2023 @02:13PM (#35805728)

Despite the lack of examples, it still completes trivial clojure like "(defn connect [" and other lisp syntax like "(define (hello" which is promising for further refinement training on Lisp languages.

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by sitkack on Wednesday May 03, 2023 @03:55PM (#35806923)

You could take it and finetune it on a bunch of Lisps, probably cost on the order of 50-500 to do that.

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by swyx on Wednesday May 03, 2023 @04:24PM (#35807288)

if anyone from MosaicML is reading this, i’d love a guide on how to do exactly this!

[ Parent | on HN ]

Re:Replit's new Code LLM: Open Source, 77% smaller than Code... by esjeon on Thursday May 04, 2023 @01:20AM (#35811526)

I hate to admit, but Python, C, Java, and JS cover most of the modern programming. But not supporting C# sounds like a bad idea.

[ Parent | on HN ]