News for hackers. Stuff that matters.
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by ofermend on Monday May 15, 2023 @10:26PM (#35957151)
All the code generation tools, StarCoder included, still have hallucinations. In this context code that looks good, but doesn't work or has a subtle bug. How do we address that?
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by arthurcolle on Monday May 15, 2023 @10:28PM (#35957161)
Verification systems that then feed back into the models and correct hallucinations. It is slow but I think that's the only real way forward
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by DavidKarlas on Monday May 15, 2023 @11:16PM (#35957387)
Sounds same as when I started codding, hallucinated some code, compiler told me it is non-sense, and eventually I understood most of the rules... I still mess it up...
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by arthurcolle on Tuesday May 16, 2023 @10:44AM (#35962584)
If you keep it up, eventually you get good
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by theaiquestion on Monday May 15, 2023 @11:25PM (#35957433)
> All the code generation tools, StarCoder included, still have hallucinations.

This also includes humans. We "hallucinate" in very similar ways. For example mistaking localhost:8080 for localhost:8008 in a large config file. Attempting to use methods that were deprecated and no longer exist, etc.

IMO there's two ways to prevent this is - one is to make better performing models (architecture/training data/training amount/etc)

The other is the exact same as humans. Compile time tools that let it know immediately if it hallucinated, types, linting, tests, etc.

You just do it as a loop the exact same as a human. You write code, the compiler tells you that method doesn't exist, you adjust your code/consult the documents (also doable with agents).