News for hackers. Stuff that matters.
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @10:48AM (#39303316)
No, it is `n * 3`. The challenge is simple: starting from "1", we must reach "17" by combining the operations `x * 3`, `x * 2` and `x + 5`. What is embarrassing is that Gemini manages to read the formal jargon and understand the challenge just fine. Yet it fails to combine these operations to get from "1" to "17", which is something most 10 yo kids would be able to do.
Re:Gemini Ultra Released by qsort on Thursday February 08, 2024 @10:55AM (#39303421)
Yeah, seems like I got that right. That graduate-level course must have worked. But then:

    (f (g (h (g z))))
results in:

    ((((1 * 2) + 5) * 2) * 3) = ... not 17?
while it would work if the type of f was corrected.

Or, again, am I missing something?

Re:Gemini Ultra Released by pierrebai on Thursday February 08, 2024 @11:25AM (#39303898)
You are right and I had the same reaction. The correct answer should have been: (f (h (g (h z)))) AKA ((1 + 5) * 2) + 5.

Is it not ironic that the supposedly test of AGI is flawed and its human designer fail to see it and denied it when presented with facts. Maybe the test designer is hallucinating just as much as those LLM? :)

Re:Gemini Ultra Released by scarmig on Thursday February 08, 2024 @12:03PM (#39304413)
> The correct answer should have been: (f (h (g (h z)))) AKA ((1 + 5) * 2) + 5.

Isn't that (h(g(h z)))?

And, FWIW, at least in my test, Gemini gets that in its final answer, though it failed in the two other drafts:

https://g.co/gemini/share/c922e7ef62aa

ChatGPT sputters:

https://chat.openai.com/share/25abbf47-2ed4-4635-a351-90a9a6...

(ETA more Gemini testing suggests its correct answer was a one-off)

Re:Gemini Ultra Released by mahogany on Thursday February 08, 2024 @11:29AM (#39303948)
It's pretty amusing and it is not the first time I've seen this. Random example: https://news.ycombinator.com/item?id=38387168

It's a little scary that it can be so hard to evaluate the correctness of these LLMs even when we are paying close attention and looking for mistakes. Or maybe the scary part is that we can become biased when we want to believe.

Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:33AM (#39304002)
Oh fuck. Well, in my defense, nobody is claiming I'll design fusion reactors and cure cancer by 2027
Re:Gemini Ultra Released by og_kalu on Thursday February 08, 2024 @04:44PM (#39308149)
No, but we at least acknowledge you a general intelligence like all humans. I'm not sure when agi - artificial general intelligence began to mean anything other than artificial and generally intelligent.

agi may as well be God, the bars some people have.

Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:29AM (#39303951)
Oh, lol, you're right. Seems like I'm dumber than both AIs. GPT-4 mixed up `h` and `f`, so it also got it wrong, so this is a draw and both AIs (and, apparently, myself) are terrible at reasoning. Guess we're not curing cancer with computers anytime soon :')
Re:Gemini Ultra Released by motoxpro on Thursday February 08, 2024 @05:42PM (#39308813)
so much for "Anyone familiar with the syntax / jargon should be able to answer this specific problem in ~5 seconds of thinking, though. And I mean it, even a 10yo kid should..."