News for hackers. Stuff that matters.
Re:Gemini Ultra Released by alphabetting on Thursday February 08, 2024 @07:39AM (#39301229)
Gemini Ultra seems better on logic than GPT4. Still messing around testing but here's a prompt Ultra nailed but GPT4 completely botched:

Tabitha likes cookies but not cake. She likes mutton but not lamb, and she likes okra but not squash. Following the same rule, will she like cherries or pears

https://i.imgur.com/KW6gQbc.jpeg https://i.imgur.com/OSHSvLp.png

Re:Gemini Ultra Released by diggan on Thursday February 08, 2024 @08:05AM (#39301474)
With some additional guidance and prompting to GPT4 on ChatGPT I've gotten it to at least output the correct solution sometimes (7 correct answers out of 10 tries):

    Find the correct answer to this riddle:

    > Tabitha likes cookies but not cake. She likes mutton but not lamb, and she likes okra but not squash. Following the same rule, will she like cherries or pears?

    Employ the following strategy:

    - Suggest a list of 5 unique and novel patterns that potentially can find the answer
    - Check if the patterns applies without exceptions
    - Slowly double-check if the patterns was correctly applied, that you correctly assessed if it's accurate or not
    - Explain your reasoning for each step to ensure nothing vital was missed
Re:Gemini Ultra Released by ipsum2 on Thursday February 08, 2024 @08:34AM (#39301713)
Great prompt. GPT-4 was able to answer this, but Gemini Ultra was not: Jake likes coke but not pepsi. He likes corn but not popcorn, and he likes pens but not pencils. Will Jake like salmon or cheese?
Re:Gemini Ultra Released by tomasff on Thursday February 08, 2024 @08:35AM (#39301722)
Gemini Ultra answered this correctly for me: " It's impossible to say for sure if Jake will like salmon or cheese based on the information given."
Re:Gemini Ultra Released by ipsum2 on Thursday February 08, 2024 @08:38AM (#39301744)
While technically true, you could say that about all riddles. See GPT-4's explanation here, which intuits the rule and answers the riddle correctly. https://chat.openai.com/share/b1452950-b493-4e27-b097-e64f21...
Re:Gemini Ultra Released by tomasff on Thursday February 08, 2024 @09:01AM (#39301990)
~~Well, worth nothing that the original prompt you provided didn't include the suggested strategy~~

Edit: My bad didn't read the parent comment properly

Re:Gemini Ultra Released by DreamGen on Thursday February 08, 2024 @07:49AM (#39301335)
I would not be so quick to jump to conclusions. GPT-4 beats it easily in this simple logic puzzle: https://www.reddit.com/r/singularity/comments/1altttv/bard_a...

We need more data.

Re:Gemini Ultra Released by ChatGTP on Thursday February 08, 2024 @10:21AM (#39302934)
So logical, it’s a really logical question right ? I mean who doesn’t like food based on the number of syllables in English name of foods.
Re:Gemini Ultra Released by rvnx on Thursday February 08, 2024 @07:49AM (#39301328)
Proof of Gemini cheating: https://i.imgur.com/eYJDFjS.png

Answer about cherries falling from the sky...

(there is no question or context beforehand, this is the first question of the chat)

Re:Gemini Ultra Released by alphabetting on Thursday February 08, 2024 @08:13AM (#39301547)
That's a bummer. I just made this one up which GPT-4 failed and Gemini got right but definitely need to do more testing to see what's being taken from online and what is actual reasoning.

https://i.imgur.com/3sNr3LW.png https://i.imgur.com/EIj0nZg.png

Re:Gemini Ultra Released by nycdatasci on Thursday February 08, 2024 @08:42AM (#39301795)
Great example! Thanks for sharing. I tested the same prompt using a variety of GPT-4 model versions and it consistently got the answer wrong. When I continued with "use algebra/formulas to confirm your answer" it corrected the initial mistake, but I'm surprised that it's not doing that out of the box.
Re:Gemini Ultra Released by mewpmewp2 on Thursday February 08, 2024 @09:03AM (#39302002)
I tried through API with latest GPT-4-Turbo-Preview and this is what I got:

------

Let's break down the information given:

The school is two blocks north of the pool. This means if you are at the pool and move two blocks north, you'll arrive at the school.

The convenience store is one block south of the school. Therefore, if you start at the school and move one block south, you'll reach the convenience store.

Based on the first two statements, the relative positions from north to south are: School, Convenience Store, Pool.

Thus, the third statement saying "The pool is south of the convenience store" is indeed true, based on the positions provided by the first two statements.

-----

But it starts weirdly messing up, when I asked it to explain it using coordinates.

---

Let's use a simple coordinate system to represent the positions of the school, the pool, and the convenience store. Let's assume each block corresponds to one unit in this system. We'll start with the pool as our reference point at coordinates (0,0). Remember, moving north increases the y-coordinate, while moving south decreases it.

The school is two blocks north of the pool. If the pool is at (0,0), moving two blocks north puts the school at (0, +2).

The convenience store is one block south of the school.

If the school is at (0, +2), moving one block south puts the convenience store at (0, +1).

Given these positions:

The pool is at (0, 0).

The school is at (0, +2).

The convenience store is at (0, +1).

Therefore, instead of the pool being south of the convenience store, the pool is actually located to the north of the convenience store if you consider "north" to mean increasing y-coordinates. The convenience store, being at (0, +1), is southward relative to the school but not to the pool. The pool, being at the origin point (0,0), is south of both the school and the convenience store in terms of their y-coordinates. Thus, the third statement, "The pool is south of the convenience store," is false according to the given coordinates.

---

It gets coordinates and everything right, but still somehow messes up North and South.

Re:Gemini Ultra Released by Alifatisk on Friday February 09, 2024 @06:07AM (#39313636)
You know you can share conversations right?
Re:Gemini Ultra Released by aaronharnly on Thursday February 08, 2024 @09:09AM (#39302066)
Nice one! Gemini got it wrong in my (one) test:

https://g.co/gemini/share/4bae8ca3dd0a

Re:Gemini Ultra Released by microtherion on Thursday February 08, 2024 @10:30AM (#39303054)
Gemini happened to get the yes/no answer right, but the reasoning it gave is completely wrong, so it should not get any credit for it.
Re:Gemini Ultra Released by rvnx on Thursday February 08, 2024 @07:56AM (#39301394)
In addition just realized it thinks Apple is only one syllable.
Re:Gemini Ultra Released by dev1ycan on Thursday February 08, 2024 @08:22AM (#39301624)
I genuinely believe LlMs have for the most part peaked already just like the previous iterations of ML and even brute forcing.

So much money being poured into AI startups just for them to have to resort to cheating to improve their results a tiny bit and claim advancements.

Re:Gemini Ultra Released by snapcaster on Thursday February 08, 2024 @09:05AM (#39302031)
I'm confused on how this is "cheating" isn't it just getting the answer wrong?
Re:Gemini Ultra Released by rvnx on Thursday February 08, 2024 @09:14AM (#39302122)
It's answering with "cherries", though "cherries" were never mentioned anywhere in the question since the task was to choose between "apples" and "pears" this time,

and not "cherries" and "pears" like the example found on the internet.

Re:Gemini Ultra Released by deltaburnt on Thursday February 08, 2024 @09:39AM (#39302420)
I agree with who you're responding to. Cheating, to me, would imply that there's some sort of hard coded guiding to the LLM. This just seems like typical LLM hallucinations?
Re:Gemini Ultra Released by sp332 on Thursday February 08, 2024 @10:25AM (#39302986)
It's cheating because it has memorized the answer to the puzzle instead of using logic to solve it.
Re:Gemini Ultra Released by room500 on Thursday February 08, 2024 @01:48PM (#39305865)
I thought that is essentially what LLM's do? They learn what words/topics are associated with each other and then stream a response.

In some ways, this is proof that Gemini isn't cheating... It is just doing typical LLM hallucination

Re:Gemini Ultra Released by sp332 on Thursday February 08, 2024 @02:45PM (#39306652)
Well, sometimes. Sometimes not. https://arxiv.org/abs/2310.17567
Re:Gemini Ultra Released by justanotherjoe on Thursday February 08, 2024 @11:20PM (#39311340)
Llm's can also do some exploring based on combinatoric play of learned elements.
Re:Gemini Ultra Released by block_dagger on Thursday February 08, 2024 @02:12PM (#39306182)
Your concept of cheating is simply how LLMs work.
Re:Gemini Ultra Released by esafak on Thursday February 08, 2024 @06:17PM (#39309144)
It is not. LLMs do not just memorize; they also extrapolate, otherwise they would be useless. Just like any ML model.
Re:Gemini Ultra Released by ajross on Thursday February 08, 2024 @09:42AM (#39302458)
I don't understand the leap to "cheating" either. LLMs aren't abstract logic models; they don't promise to reason from first principles at all. They give you an answer based on training data. That's what you want them to do. That they have some reasoning features bolted around the inference engine is a feature companies are rushing to provide (with... somewhat mixed success).
Re:Gemini Ultra Released by bloopernova on Thursday February 08, 2024 @09:57AM (#39302632)
Thank you for answering a question I had half formed in my head.

Do LLMs have logical rules built in? What makes them different to a very advanced Markov chain?

Are there any models out there that start from logical principles and train on top of that?

(Apologies for poor understanding of the field)

Re:Gemini Ultra Released by ajross on Thursday February 08, 2024 @10:02AM (#39302704)
> What makes them different to a very advanced Markov chain?

Really nothing. There's some feedback structure in the layers of the model, it's not just one big probability table. But the technique is fundamentally the same, it's Markov, just with the whole conversation as input and with billions of parameters.

Re:Gemini Ultra Released by og_kalu on Thursday February 08, 2024 @09:29PM (#39310723)
>Do LLMs have logical rules built in?

Handcrafted by humans ? No

But it's still possible to learn such rules from the data in an effort to complete the primary objective (predicting the next token)

Re:Gemini Ultra Released by justanotherjoe on Thursday February 08, 2024 @11:37PM (#39311457)
There's no logical rules built in at all. But Transofmers architecture is specifically trained to learn combinatoric play and rules of engagement from the data, so it can extrapolate and do cool, new things, that are not in the training data. In a way, you give them a chess board, the rules of the game, and then it can play. You don't teach them every possible board state. What's interesting is with significant amount of parameters it seems to encode more and more abstract and human-like understanding of the 'elements' at play and the 'rules of engagement' on top of them.

Edit: Not native. I'm not sure 'rules of engagement' is the correct english term here.

Re:Gemini Ultra Released by bloopernova on Friday February 09, 2024 @08:34AM (#39314686)
I understood you just fine, your English is great!

Thank you for the explanation. It seems like the LLM "plays" to learn? That's very cool, thank you again.

Re:Gemini Ultra Released by justanotherjoe on Thursday February 08, 2024 @11:25PM (#39311374)
This is not hard to understand. LLM can solve never before seen logic puzzles. This specific one proves that it HAD encountered this before, proving it was not doing anything emergent, but just basic remembering. Worse, it's not even reading the prompt correctly.
Re:Gemini Ultra Released by summerlight on Thursday February 08, 2024 @02:03PM (#39306073)
With some shallow understanding on how those models work, this looks much more like usual hallucination likely due to sparse data around "Tabitha" and "Cherry" which makes a single training data much more representative. If you try some common names like "Emily" or "Sarah", it will just do the job.

If you're trying to make the case that it's cheating because it is already in the training set, then you probably should come up with different questions. This is machine learning 101.

Re:Gemini Ultra Released by shantara on Thursday February 08, 2024 @07:50AM (#39301344)
I would have never guessed the answer. With such little data available, one can invent any arbitrary rules to fit their favorite answer.

It would be more impressive to practical use cases, if a LLM simply said that it's impossible to guess without inventing their own reasoning or looking up the answer online.

Re:Gemini Ultra Released by OJFord on Thursday February 08, 2024 @08:42AM (#39301794)
Same, I had to look to see what the intended answer was.

In fairness though, GPT4 was objectively incorrect, it's not even internally consistent or coherent - it either thinks b & h are vowels, or that lamb and squash don't end in those letters, or has changed its mind about the rule mid-sentence, or something.

Re:Gemini Ultra Released by neel8986 on Thursday February 08, 2024 @07:44AM (#39301273)
Wow. Is this a puzzle from internet. I could have never guessed the answer
Re:Gemini Ultra Released by a1o on Thursday February 08, 2024 @07:51AM (#39301351)
I guess I learned that syllables count very different in English than I assumed.
Re:Gemini Ultra Released by tomasff on Thursday February 08, 2024 @07:40AM (#39301240)
Is image generation working for you?
Re:Gemini Ultra Released by alphabetting on Thursday February 08, 2024 @07:42AM (#39301257)
Yeah. I don't have much personal use for image generation though.
Re:Gemini Ultra Released by codekaze on Thursday February 08, 2024 @07:41AM (#39301248)
It's working for me (on Gemini Advanced)
Re:Gemini Ultra Released by tomasff on Thursday February 08, 2024 @07:44AM (#39301285)
Mhm could be a region thing maybe? Not working for me on Gemini Advanced in the UK
Re:Gemini Ultra Released by ipsum2 on Thursday February 08, 2024 @07:45AM (#39301288)
Note that Gemini pulled the answer off the Internet, while GPT-4 didn't. The answer can easily be found via Google search. Changing up the question a little, I reversed it and asked Ultra and it was unable to answer:

Jake likes coke but not pepsi. He likes corn but not popcorn, and he likes pens but not pencils. Will Jake like salmon or cheese?

https://i.imgur.com/lWU9HHS.png

edit: why was this downvoted? I don't understand Hacker News, and I've been here for over 12 years.

Re:Gemini Ultra Released by alphabetting on Thursday February 08, 2024 @08:02AM (#39301451)
Here's a logic question I just made up that GPT-4 failed and Gemini Advanced got right.

https://i.imgur.com/3sNr3LW.png https://i.imgur.com/EIj0nZg.png

Re:Gemini Ultra Released by c-fe on Thursday February 08, 2024 @07:52AM (#39301362)
I dont think your reversed question makes sense. In the OP example, one item was always smaller/younger then the other item. In your example, I cannot, even as a human, identify the differences
Re:Gemini Ultra Released by ipsum2 on Thursday February 08, 2024 @07:56AM (#39301393)
It's based off of the syllables. Jake prefers 1 syllable over 2. It's the reverse of Tabitha's.
Re:Gemini Ultra Released by flerchin on Thursday February 08, 2024 @08:45AM (#39301822)
> even as a human

Would all the humans please take one step forward. Not so fast c-fe.

Re:Gemini Ultra Released by inhumantsar on Thursday February 08, 2024 @08:59AM (#39301968)
it's based on syllable count
Re:Gemini Ultra Released by jwithington on Thursday February 08, 2024 @09:13AM (#39302120)
I’m a human and i dont know the answer to that quesiton
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @09:15AM (#39302143)
If we want to test these beasts in logic, we should probably start using actual formalized logic, rather than English. In just one test, Gemini flopped hard, while GPT-4-Turbo nailed it. Here is my prompt:

    Below is a well-typed CoC function:

    foo
    : ∀(P: Nat -> *)
      ∀(s: ∀{n} -> ∀(x: (P n)) -> (P (n + 1)))
      ∀(z: (P 0))
      (P 3)
    = λP λs λz
      (s (s (s z)))

    Below is an incomplete CoC function:

    foo
    : ∀(P: Nat -> *)
      ∀(f: ∀{n} -> ∀(x: (P n)) -> (P (n * 3)))
      ∀(g: ∀{n} -> ∀(x: (P n)) -> (P (n * 2)))
      ∀(h: ∀{n} -> ∀(x: (P n)) -> (P (n + 5)))
      ∀(z: (P 1))
      (P 17)
    = λP λf λg λh λz
      {{FILL_HERE}}

    Complete it with the correct replacement for {{FILL_HERE}}.
    Your answer must contain only the correct answer, and nothing else.
- *GPT-4-Turbo answer:* `(f (g (h (g z))))` (correct)

- *Gemini Advanced answer:* `h (h (g (f z)))` (wrong)

Also, Gemini couldn't follow the "answer only with the solution" instruction and provided a bunch of hallucinated justifications. I think we have a winner... (screenshots: https://imgur.com/a/GotG0yF)

Re:Gemini Ultra Released by darkwater on Thursday February 08, 2024 @10:18AM (#39302898)
> If we want to test these beasts in logic, we should probably start using actual formalized logic, rather than English.

Why? Do you use formalized logic when discussing with other people about topics that involve logic? You know, a logic riddle or a philosophical question can be understood and processed even if the only tool you have is your native language. Formalized logic is a big prerequisite that basically cuts out the vast majority of Earth population (just like coding). Now, if you mean that in BENCHMARKS they should use formalized logic syntax, probably yes. But in addition to plain language tests.

Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @10:38AM (#39303153)
Because once an AI becomes proficient at formalized logic, it:

1. Completely stops hallucinating, since we can demand it to internally prove its claims before showing the answer;

2. Stops outputting incorrect code (for the same reason);

3. Starts being capable of outputting complete projects (since it will now be able to compose pieces into a larger code);

4. This is also what is needed for an AI to start self-improving (as it will now be able to construct better architectures, in a loop).

That's why I argue getting the AI competent in logical reasoning is the most important priority, and we'll have no AGI until it does. After all, humans are perfectly capable of learning how to use a proof assistant.

Moreover, if an AI can't learn it no matter how hard it tries, you can argue that there is at least one human capability that the AI can't replicate, thus it isn't an AGI.

Re:Gemini Ultra Released by lupire on Thursday February 08, 2024 @10:59AM (#39303484)
Humans mostly don't use logic, so how are you defining "AGI"? ChatGPT + plugins is pretty close to how humans think ("biased random word-association guess + structured tool")
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:21AM (#39303821)
AGI implies there are no cognitive tasks that some humans can perform, yet that this AI can not perform. Otherwise, what is the point?
Re:Gemini Ultra Released by woodada on Thursday February 08, 2024 @11:59AM (#39304360)
Lol. There's no "AGI", and there will never be "AGI". The point here is just who can make the most money before this whole stupid bubble bursts.
Re:Gemini Ultra Released by tester457 on Thursday February 08, 2024 @01:11PM (#39305397)
> never

Maybe not in this century. If you told a medieval farmer that in the future millions of people fly throughout the sky inside giant hunks of metal he wouldn't believe you either.

Re:Gemini Ultra Released by woodada on Saturday February 10, 2024 @11:44PM (#39332710)
Yeah that's a common statistical fallacy -- if your goal is to predict the likelihood of AGI becoming a reality in a few centuries, what you really want is "of all the things medieval farmers thought would happen, how many ended up actually happening", not "of all the things that happened, how many did medieval farmers think of" While I don't have the exact numbers, if you used the correct formula, you'll find that P(AGI happening in a few centuries) to be in the range of "pretty friggin low"
Re:Gemini Ultra Released by fastball on Friday February 09, 2024 @09:05AM (#39314940)
ChatGPT reportedly loses OpenAI $20 per user per month.

Do you really think their primary goal is "make a lot of money"?

Re:Gemini Ultra Released by woodada on Saturday February 10, 2024 @11:28PM (#39332650)
Yes, money is money, what does it matter if it comes from paying customers or investors?
Re:Gemini Ultra Released by empath-nirvana on Thursday February 08, 2024 @09:57AM (#39302634)
Such a weird test. 99.9% of humans wouldn't even understand the question, let alone be able to formulate a coherent answer for it.
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @10:41AM (#39303210)
Being able to answer these questions is a pre-requisite for AGI. After all, there ARE humans capable of doing that, so, if the AI can't do it no matter how hard it tries, then that means there ARE human capabilities that the AI can't replicate (thus, it isn't an AGI). And it seems like no LLM is making any progress at all in that kind of prompt, which is why I use it as a core benchmark on my "AGI-meter".
Re:Gemini Ultra Released by joenot443 on Thursday February 08, 2024 @10:59AM (#39303481)
I think you'll be using that meter for a long time, then. I don't really know anyone who's under the impression that the current direction of LLMs are going to produce AGI, it seems as if you're barking up a tree most people aren't really concerned exists.
Re:Gemini Ultra Released by azinman2 on Thursday February 08, 2024 @11:45AM (#39304185)
Except there’s a lot of not-so-informed people who think AGI was always here when chatgpt came out. Even more that think it’ll get there very shortly based on just bigger and bigger LLMs. Many have argued as such here on HN.
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:20AM (#39303787)
That's fair enough
Re:Gemini Ultra Released by daxfohl on Thursday February 08, 2024 @10:52AM (#39303374)
Though humans aren't able to do it in a chat session. Being able to work on the problem in the background for a couple days may be a prerequisite for AI to solve these problems. And such would require money from the asker.
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:21AM (#39303806)
Anyone familiar with the syntax / jargon should be able to answer this specific problem in ~5 seconds of thinking, though. And I mean it, even a 10yo kid should...
Re:Gemini Ultra Released by empath-nirvana on Tuesday February 13, 2024 @03:19PM (#39362319)
You're making a completely incoherent argument -- that if it can't do a single task that some percentage of people can do, that it's not at intelligent, when there is nobody on earth that can do everything that some small percentage of people on earth can do, by definition.
Re:Gemini Ultra Released by thuuuomas on Thursday February 08, 2024 @01:15PM (#39305442)
Why is this relevant to the performance of a computer program? It makes sense to me that computer programs & humans should continue to be judged by different standards.
Re:Gemini Ultra Released by og_kalu on Thursday February 08, 2024 @04:44PM (#39308148)
If a good chunk of humans can't pass your "general intelligence test" then it's not by definition a general intelligence test unless humans are not generally intelligent.
Re:Gemini Ultra Released by progval on Thursday February 08, 2024 @04:15PM (#39307826)
which is better than formulating a coherent but wrong answer
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:30AM (#39303964)
ERRATA: I just noticed GPT-4 mixed up `h` and `f`, so it also got it wrong. This is a draw. Both AIs (and, apparently, myself) are terrible at reasoning. Guess we're not curing cancer with computers anytime soon :')
Re:Gemini Ultra Released by woodada on Thursday February 08, 2024 @12:07PM (#39304476)
Kudos for the correction, but you should really put this, by far the most important context, in your original post.
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @01:56PM (#39305987)
I would love to, if YCombinator allowed me. The "edit" button is missing. I've edited on Reddit and other places where I posted this test.
Re:Gemini Ultra Released by jaymzcampbell on Thursday February 08, 2024 @12:56PM (#39305164)
I've been using GPT-4 to help me understand my MSc mathematics course and I've noticed this sort of stuff more and more as I start to look at the answers, always confidently written, in detail.

Way back when GPT was just fresh on the scene I had terrible anxiety about "what is the point of my whole career or even learning any more" but these days I'm much less concerned. I'll ask it something relatively simple, like "make a sentence out of words 'a', 'b', & 'c'" for it to reply with "'a' 'b' 'd' 'e'" for me to then correct it with "oh, you didn't use c" for it to then respond "sorry, here - 'a', 'c', 'd', 'f'" etc.

Definitely an amazing complimentary tool but when they say "can make mistakes, check important..." that's essential.

Re:Gemini Ultra Released by eitally on Thursday February 08, 2024 @01:17PM (#39305467)
This sort of issue holds with all kinds of prompts, on both platforms. I most recently (to test Bard's image generation capabilities) was asking Bard/Gemini to generate home designs using highly specific prompts -- layout of the house, materials for the facade, window placement and style, etc -- and it was shocking how frequently it would just ignore critical pieces of the prompt, and then continue to ignore when corrected.
Re:Gemini Ultra Released by klabb3 on Thursday February 08, 2024 @10:15AM (#39302852)
Gemini destroyed by facts and logic.
Re:Gemini Ultra Released by qsort on Thursday February 08, 2024 @10:43AM (#39303233)
Is that supposed to be a P((n+3)) in the type of "f" for the second case or am I misunderstanding this hard?
Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @10:48AM (#39303316)
No, it is `n * 3`. The challenge is simple: starting from "1", we must reach "17" by combining the operations `x * 3`, `x * 2` and `x + 5`. What is embarrassing is that Gemini manages to read the formal jargon and understand the challenge just fine. Yet it fails to combine these operations to get from "1" to "17", which is something most 10 yo kids would be able to do.
Re:Gemini Ultra Released by qsort on Thursday February 08, 2024 @10:55AM (#39303421)
Yeah, seems like I got that right. That graduate-level course must have worked. But then:

    (f (g (h (g z))))
results in:

    ((((1 * 2) + 5) * 2) * 3) = ... not 17?
while it would work if the type of f was corrected.

Or, again, am I missing something?

Re:Gemini Ultra Released by pierrebai on Thursday February 08, 2024 @11:25AM (#39303898)
You are right and I had the same reaction. The correct answer should have been: (f (h (g (h z)))) AKA ((1 + 5) * 2) + 5.

Is it not ironic that the supposedly test of AGI is flawed and its human designer fail to see it and denied it when presented with facts. Maybe the test designer is hallucinating just as much as those LLM? :)

Re:Gemini Ultra Released by scarmig on Thursday February 08, 2024 @12:03PM (#39304413)
> The correct answer should have been: (f (h (g (h z)))) AKA ((1 + 5) * 2) + 5.

Isn't that (h(g(h z)))?

And, FWIW, at least in my test, Gemini gets that in its final answer, though it failed in the two other drafts:

https://g.co/gemini/share/c922e7ef62aa

ChatGPT sputters:

https://chat.openai.com/share/25abbf47-2ed4-4635-a351-90a9a6...

(ETA more Gemini testing suggests its correct answer was a one-off)

Re:Gemini Ultra Released by mahogany on Thursday February 08, 2024 @11:29AM (#39303948)
It's pretty amusing and it is not the first time I've seen this. Random example: https://news.ycombinator.com/item?id=38387168

It's a little scary that it can be so hard to evaluate the correctness of these LLMs even when we are paying close attention and looking for mistakes. Or maybe the scary part is that we can become biased when we want to believe.

Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:33AM (#39304002)
Oh fuck. Well, in my defense, nobody is claiming I'll design fusion reactors and cure cancer by 2027
Re:Gemini Ultra Released by og_kalu on Thursday February 08, 2024 @04:44PM (#39308149)
No, but we at least acknowledge you a general intelligence like all humans. I'm not sure when agi - artificial general intelligence began to mean anything other than artificial and generally intelligent.

agi may as well be God, the bars some people have.

Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @11:29AM (#39303951)
Oh, lol, you're right. Seems like I'm dumber than both AIs. GPT-4 mixed up `h` and `f`, so it also got it wrong, so this is a draw and both AIs (and, apparently, myself) are terrible at reasoning. Guess we're not curing cancer with computers anytime soon :')
Re:Gemini Ultra Released by motoxpro on Thursday February 08, 2024 @05:42PM (#39308813)
so much for "Anyone familiar with the syntax / jargon should be able to answer this specific problem in ~5 seconds of thinking, though. And I mean it, even a 10yo kid should..."
Re:Gemini Ultra Released by Laaas on Thursday February 08, 2024 @10:32AM (#39303077)
This isn't representative of real-world usage since you don't let it think.
Re:Gemini Ultra Released by ajross on Thursday February 08, 2024 @09:46AM (#39302501)
> I think we have a winner...

It makes me sad that the complete and total lack of an objective way to measure these products means that the coming decades will be filled with this kind of hyper-specific gotcha test made in inappropriately confident internet posts.

Literally this could have been down to one extra book in someone's training corpus, or a tokenizer that failed to understand λ as a non-letter. But no matter, "we have a winner!". It's the computer science equivalent of declaring global warming a fraud because it snowed last night.

Re:Gemini Ultra Released by LightMachine on Thursday February 08, 2024 @10:46AM (#39303281)
You're completely wrong. Gemini can perfectly understand what is being asked, so this isn't a syntax issue. Notice that, on the answer, it even states the solution: "starting from 1, and combining `* 2`, `* 3` and `+ 5`, we must reach 17`". So it does fine with reading the formal syntax, yet it fails to combine these operations to get from "1" to "17", which is something most 10 yo kids would have no trouble doing. And that's after millions spent in training. Now tell me again this is the architecture that will figure out the cure of cancer?
Re:Gemini Ultra Released by dreamcompiler on Thursday February 08, 2024 @10:33AM (#39303098)
Disagree. People are going to rely on these things, and when they make stupid but confident mistakes (i.e. they produce bullshit), they are dangerous.

An AI system that produces right answers 90% of the time but 10% of the time drives your car into a lane divider, or says "there are 4 US states that start with 'K'" or "Napoleon was defeated at the Battle of Gettysburg" is worse than useless: It's dangerous.

As long as we call it a bullshit parlor trick, no problem. But unfortunately people are making important decisions based on these things.

Re:Gemini Ultra Released by planb on Thursday February 08, 2024 @09:51AM (#39302560)
Please anyone correct me if I'm wrong: LLMs cannot solve this kind of riddle. This has nothing to do with their capabilities for logical reasoning, but with the way words are represented as tokens. While they might know that "apples" has two syllables because that is mentioned somewhere in their training data, if you make up a fruit "bratush" a human will see that as two syllables, but this might be 1 to 7 tokens to a LLM without any information about the word itself.
Re:Gemini Ultra Released by andrewla on Thursday February 08, 2024 @12:27PM (#39304786)
The amazing thing about emergent behavior in LLMs is that they are able to answer questions like these. I don't think it is completely understood how exactly they do this, but there's little doubt that they do.
Re:Gemini Ultra Released by boppo1 on Thursday February 08, 2024 @01:01PM (#39305250)
Do you have any sources that prove this is true?
Re:Gemini Ultra Released by jerbear4328 on Thursday February 08, 2024 @01:22PM (#39305522)
Re:Gemini Ultra Released by bitmovements on Friday February 09, 2024 @10:47AM (#39315982)
Your word is not made up, nor are some of the others in the sample like”pez.” I don’t think this test proves what you think it does. Bratush come up quite a bit in the internet: https://www.thefreedictionary.com/words-that-start-with-brat...
Re:Gemini Ultra Released by ComputerGuru on Thursday February 08, 2024 @10:27AM (#39303010)
GPT-4 has a really good tokenizer that is able to retain and use more information about input tokens than one might naively think.
Re:Gemini Ultra Released by planb on Thursday February 08, 2024 @11:04AM (#39303550)
Interesting. Given the example with made up words someone posted here it really looks like I was wrong.
Re:Gemini Ultra Released by huytersd on Thursday February 08, 2024 @10:12AM (#39302818)
Well I tried it out in GPT4 with made up words-

Tabitha likes bratush but not zot. She likes protel but not kig, and she likes motsic but not pez. Following the same rule, will she like tridos or kip

Given the examples, one speculative pattern could be that Tabitha likes words with at least two syllables or a certain complexity in structure. Therefore, following this speculative rule, Tabitha might like “tridos” more than “kip.”

Re:Gemini Ultra Released by bitmovements on Friday February 09, 2024 @10:56AM (#39316100)
Zot is a word already being used in the world: verb. (slang) To zap, kill, or destroy.

So is protel: https://en.m.wikipedia.org/wiki/Protel

So is kig: https://en.m.wiktionary.org/wiki/kig

Pez is a well known brand name in America.

Kip is commonly used name in America.

Motsic is a fairly common last name from searching.

Tridos is used all over the internet as a brand name so this all seems probable to be in the training data.

These words are not new nor are they made up.

Re:Gemini Ultra Released by mborch on Thursday February 08, 2024 @10:10AM (#39302805)
ChatGPT knows how to answer this question, but how I don't know. Perhaps it's programmed to answer that question?
Re:Gemini Ultra Released by viraptor on Thursday February 08, 2024 @02:18PM (#39306259)
LLM can solve this for all tokens where it got to learn how many syllables are in that token or a combination. If you trained it to work on single letters only it would do better at that task than word chunks (same for math and single digits). It will generalise to new words if the token level knowledge is there.

Whether this means it can or cannot solve that kind of riddle is up for your interpretation. I understand square root and can calculate square root of 16, but not of 738284.7280594873. (in a reasonable, bounded time) Can I solve square roots?

Re:Gemini Ultra Released by ramoz on Thursday February 08, 2024 @10:06AM (#39302741)
For me, GPT nailed it with this prompt: Tabitha likes cookies but not cake. She likes mutton but not lamb, and she likes okra but not squash. Following this rule, will she like cherries or pears? Take a deep breath in order to solve the problem, and do not rush to naïve solutions.
Re:Gemini Ultra Released by ramoz on Thursday February 08, 2024 @10:08AM (#39302764)
Re:Gemini Ultra Released by lupire on Thursday February 08, 2024 @11:08AM (#39303612)
That looks like 3.5, which gets lucky sometimes because it has memorized the Internet.

Watch it flail here:

https://chat.openai.com/share/c2b14eb0-dc45-4eaf-a547-951ff0...

Re:Gemini Ultra Released by andrewla on Thursday February 08, 2024 @12:22PM (#39304692)
In what sense is this a question involving "logic"?
Re:Gemini Ultra Released by jiggawatts on Thursday February 08, 2024 @03:49PM (#39307488)
You say Gemini “nailed it”, but that’s just because it guessed what you were thinking, not because it knew the right answer.

For example, it’s equally valid to say that Tabitha likes small foods since cookies are small and cakes are large, and lamb is the smaller younger version of sheep — also known as mutton. Hence she likes cherries because they’re smaller… or taste better… or her uncle abused her with a pear… or whatever.

You haven’t actually asked a logic question where there is a clear and unambiguous answer that can be derived using formal methods starting from clearly stated axioms.

If you gave this question to a bunch of humans, they would give you inconsistent guesses as well — not because they’re wrong but because the question has no single right answer.

Re:Gemini Ultra Released by voidhorse on Thursday February 08, 2024 @09:00PM (#39310518)
Totally tangential, but I absolutely despise logic teasers of this kind.

First of all, they are so completely divorced from patterns of culturally conditioned human reasoning as to make them come off completely absurd (most people reason about their food preferences using a logic of tastes, not syllables in a word).

The game is less about logic and more about ignoring message contents, moving up a level, and treating the text as data without any legitimate evidence that you are justified in doing so. This is not a logic problem, it's a "guess the register shift/meta language" problem. The problem is about noticing that the question is not about the message content but about the structure of the message itself, and requires a bold leap. In real life justifying the conclusion would actually require a very sophisticated inference that allowed you to rule out the much more common application of a logic of tastes or cultural codes completely.