News for hackers. Stuff that matters.
Re:OpenDevin: An Open Platform for AI Software Developers as... by threeseed on Sunday August 11, 2024 @02:10PM (#41218162)
This assumes that the model knows it is wrong. It doesn't.

It only knows statistically what is the most likely sequence of words to match your query.

For rarer datasets e.g. I had Claude/OpenAI help out with an IntelliJ plugin it would continually invent methods for classes that never existed. And could never articulate why.

Re:OpenDevin: An Open Platform for AI Software Developers as... by popinman322 on Sunday August 11, 2024 @02:59PM (#41218542)
This is where supporting machinery & RAG are very useful.

You can auto- lint and test code before you set eyes on it, then re-run the prompt with either more context or an altered prompt. With local models there are options like steering vectors, fine-tuning, and constrained decoding as well.

There's also evidence that multiple models of different lineages, when their outputs are rated and you take the best one at each input step, can surpass the performance of better models. So if one model knows something the others don't you can automatically fail over to the one that can actually handle the problem, and typically once the knowledge is in the chat the other models will pick it up.

Not saying we have the solution to your specific problem in any readily available software, but that there are approaches specific to your problem that go beyond current methods.

Re:OpenDevin: An Open Platform for AI Software Developers as... by __loam on Sunday August 11, 2024 @04:30PM (#41219128)
This is a really complicated (and more expensive) setup that doesn't fundamentally fix any of the problems with these systems.
Re:OpenDevin: An Open Platform for AI Software Developers as... by dartos on Monday August 12, 2024 @08:08AM (#41223507)
Popular programming in a nutshell.

It’s the new pop psych.

Re:OpenDevin: An Open Platform for AI Software Developers as... by segfaltnh on Monday August 12, 2024 @10:22PM (#41231557)
Yep when I read stuff like this I think, "nah I'll just write the damn code." Looking forward to being replaced by a robot, myself.
Re:OpenDevin: An Open Platform for AI Software Developers as... by threeseed on Sunday August 11, 2024 @06:11PM (#41219650)
It doesn't make sense that the solution here is to put more load on the user to continually adjust the prompt or try different models.

I asked Claude and OpenAI models over 30x times to generate code. Both failed every time.

Re:OpenDevin: An Open Platform for AI Software Developers as... by thierrydamiba on Monday August 12, 2024 @10:04AM (#41224491)
If Claude and OpenAI are so useless why does every company ban it during interviews?
Re:OpenDevin: An Open Platform for AI Software Developers as... by noddingham on Monday August 12, 2024 @11:15AM (#41225401)
Do think that says more about the tools or the interview process?
Re:OpenDevin: An Open Platform for AI Software Developers as... by krageon on Tuesday August 13, 2024 @08:21AM (#41234748)
Managers make most of those decisions and they have no idea what is achievable, reasonable or even particularly likely.