News for hackers. Stuff that matters.
Re:Comparing 60 LLMs with a set of 20 prompts by 6510 on Saturday September 09, 2023 @04:01PM (#37449686)
I much enjoyed this one:

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? https://benchmarks.llmonitor.com/sally

It states that the correct answer is 1.

This is incorrect!

This one, while still kinda wrong, is the closest:

> Claude Instant v1: Sally has 0 sisters. The question provides no information about Sally having any sisters herself.

1) A sister is a female who has one or both parents in common with another individual.

2) A brother is a male who has one or both parents in common with another individual.

It takes only one shared parent to make someone Sally's brother. She will always be his sister but he can have additional sisters that share the other parent.

Sally's mum is Jane and her dad is Jim. Jane had a son with Jack. Jack had 2 daughters with Zoe. Jim had 2 sons with Mary, Mary had 2 daughters with Dan.

   Zoe      Jack        Jane      Jim        Mary        Dan
      \    /    \       /   \    /   \      /    \       /
    sis1.1&1.2   brother1   Sally   brother2&3   sis2.1&2.2   
Sally has at least zero sisters.

Also possible: Sally's mum is Jane and her dad is Jim. Jane has a son and a daughter with Jack. Jim has 2 sons and a daughter with Mary.

      Jack            Jane      Jim       Mary 
          \           /  \    /    \     /       
        sis1&brother1    Sally   brother2&3&sis2   
Sally has at most 2 sisters.

We should be this dense about the question.

Re:Comparing 60 LLMs with a set of 20 prompts by pneumonic on Sunday September 10, 2023 @12:23AM (#37452885)
Siblings and half-siblings are not the same.
Re:Comparing 60 LLMs with a set of 20 prompts by 6510 on Sunday September 10, 2023 @01:33AM (#37453178)
It's my second language. Is it not correct to call a half-brother your brother in English?

I seems very odd to say it is not a brother at all? Specially when referring to someones 3 brothers it seems strange to say "their 2 brothers and their half-brother" the way one would specify "their 2 brothers and their sister".

Re:Comparing 60 LLMs with a set of 20 prompts by 6510 on Friday September 15, 2023 @12:51AM (#37518592)
The dictionary says a brother needs one parent in common.

Therefore I'm right and everyone else is wrong. (the jokes on you everyone else!) The LLM's are just more wrong which is just a different kind of wrong.