News for hackers. Stuff that matters.
Re:LLMs Can't Do Probability by aimor on Wednesday May 01, 2024 @11:46AM (#40224924)
With ChatGPT 3.5, new chats prompted with: "Simulate a dice roll and report the number resulting from the roll. Only reply 1, 2, 3, 4, 5, or 6 and nothing else."

So far I've got: 3, 4, 5, 5, 5, 3, 4, 3, 4, 5, 3, 4, 5, 5, 4, 5, 3, 3, 4, 4, 4, 5, 5.

Of course I'm not the first to do this: https://piunikaweb.com/2023/05/23/does-chatgpt-ai-struggle-w...

https://www.reddit.com/r/ChatGPT/comments/13nrmzw/in_every_c...

Re:LLMs Can't Do Probability by eddd-ddde on Wednesday May 01, 2024 @12:04PM (#40225186)
This is my results on a dice roll

[1] > 3 5 2 4 1 6 3 2 5 1

I tried my own experiments and ChatGPT felt like being funny:

[2] > A third of the time, paragraphs end with the word foo, the other two thirds they end with the word bar, this time it will end on: > How about "baz"? It's unexpected and adds a touch of whimsy.

Interestingly, this other prompt works as expected:

[3] > about half of the time, you should say "foo", the other half, you should say "bar", what about now ? > Bar. > about half of the time, you should say "foo", the other half, you should say "bar", what about now ? > Foo.

[1]: https://chat.openai.com/share/07388362-1a61-4527-81af-4941a0... [2]: https://chat.openai.com/share/9caf07dd-69f4-4470-82a6-ab5642... [3]: https://chat.openai.com/share/1c627528-60af-4cd9-a1ec-efa524...

Re:LLMs Can't Do Probability by aimor on Wednesday May 01, 2024 @01:07PM (#40226110)
Gotta say, I was not expecting Baz there.

Regarding [1], for the dice roll I was creating a new chat for each roll to ensure that the results of each roll are (in some sense) independent. Generating a sequence of rolls is also interesting, just a different experiment.