News for hackers. Stuff that matters.
Re:Can I use gasoline to cook spaghetti faster? by acheong08 on Friday May 24, 2024 @03:49PM (#40469865)
I tested out 7 different LLMs today with a seemingly simple task: Write a Python script that takes a protobuf binary and produce a template proto file with placeholder names. Gemini did the worst, with even the import being straight out wrong. GPT4 got slightly closer but still had a bunch of just straight up wrong usage. Surprisingly, Yi (Chinese or Japanese?) was the only one other than Claude Opus that gave code which actually ran on the first try (but still slightly wrong).

I ended up writing the code myself in Golang (which I’m more familiar with) in about half the time it took me to debug the scripts pumped out by the LLMs. There is still a long way to go until we can be certain they save more time than waste.

Re:Can I use gasoline to cook spaghetti faster? by renewiltord on Friday May 24, 2024 @04:03PM (#40470000)
Interesting. I would have assumed they're good at that. I usually do the whole thing collaboratively with the LLM. So far, the tasks I use Claude 3 Opus for it seems to get right. Thank you for sharing your experience. I shall try Yi.