Re:StarCoder and StarCoderBase: 15.5B parameter models with ...
by ofermend on Monday May 15, 2023 @10:26PM
(#35957151)
All the code generation tools, StarCoder included, still have hallucinations. In this context code that looks good, but doesn't work or has a subtle bug. How do we address that?
[ Parent
| on HN ]
Re:StarCoder and StarCoderBase: 15.5B parameter models with ...
by arthurcolle on Monday May 15, 2023 @10:28PM
(#35957161)
Verification systems that then feed back into the models and correct hallucinations. It is slow but I think that's the only real way forward
Re:StarCoder and StarCoderBase: 15.5B parameter models with ...
by DavidKarlas on Monday May 15, 2023 @11:16PM
(#35957387)
Sounds same as when I started codding, hallucinated some code, compiler told me it is non-sense, and eventually I understood most of the rules... I still mess it up...
Re:StarCoder and StarCoderBase: 15.5B parameter models with ...
by arthurcolle on Tuesday May 16, 2023 @10:44AM
(#35962584)
If you keep it up, eventually you get good
Re:StarCoder and StarCoderBase: 15.5B parameter models with ...
by theaiquestion on Monday May 15, 2023 @11:25PM
(#35957433)
> All the code generation tools, StarCoder included, still have hallucinations.
This also includes humans. We "hallucinate" in very similar ways. For example mistaking localhost:8080 for localhost:8008 in a large config file. Attempting to use methods that were deprecated and no longer exist, etc.
IMO there's two ways to prevent this is - one is to make better performing models (architecture/training data/training amount/etc)
The other is the exact same as humans. Compile time tools that let it know immediately if it hallucinated, types, linting, tests, etc.
You just do it as a loop the exact same as a human. You write code, the compiler tells you that method doesn't exist, you adjust your code/consult the documents (also doable with agents).