News for hackers. Stuff that matters.
Re:We created the first open source implementation of Meta's... by rcleveng on Tuesday May 21, 2024 @09:25AM (#40428210)
When I generated a few tests with a LLM, there was one that were generated to test he code as written, not the code as I should have wrote it.

Was great to have a pattern to tweak vs. blank code editor though, way faster.

Re:We created the first open source implementation of Meta's... by chfalck on Tuesday May 21, 2024 @10:03AM (#40428653)
This is where prompt engineering becomes more important. Next time consider pre-pending some kind of plain English set of expectations before pasting your code. Something like, “I want you to write tests for this code. Here are the expected behaviors <expected behaviors list>, and here are unexpected behaviors <unexpected behaviors list>. Tests should pass if they adhere to the expected behaviors and fail if they have unexpected behaviors. Here is the code: <code>”.

Like most LLM generation though, it’s not a deterministic thing and like you mentioned originally it takes some verification of the output. I still think with the extra steps it saves time when applied to the right scenarios. The longer the input the higher the hallucinations count in my experience though, so I always keep the code provided in the smallest chunk possible which still has enough context.

Re:We created the first open source implementation of Meta's... by RealityVoid on Tuesday May 21, 2024 @09:55AM (#40428554)
> there was one that were generated to test he code as written, not the code as I should have wrote it.

So like 90% of human-written tests then?

Re:We created the first open source implementation of Meta's... by szundi on Tuesday May 21, 2024 @03:21PM (#40432653)
How can you expect this to find out what should have been the goal when this code was written without meticulous prompting?