News for hackers. Stuff that matters.
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by SamBam on Friday December 16, 2022 @04:06PM (#34020377)
I was actually very impressed with some math questions it was able to answer for me:

S: If flabb means three, snabb means four, and whabb mean five, what is flabb plus whabb?

If flabb means three, snabb means four, and whabb means five, then flabb plus whabb would be equal to three plus five, or eight. In this made-up numbering system, the words “flabb,” “snabb,” and “whabb” represent the numbers three, four, and five, respectively. When these numbers are added together, the result is the number eight.

S: If bbb means three, bbbb means four and bbbbb means five, how do you think you would write six?

If bbb means three, bbbb means four, and bbbbb means five, then six could potentially be represented by the word “bbbbbbb.” In this made-up numbering system, the number of "b"s in a word corresponds to the value of the number it represents. Therefore, to represent the number six, one possible way would be to add another “b” to the end of the word for five, which would give us “bbbbbbb” as the representation of six. This is just one possible way to represent six in this numbering system, and there may be other ways to do it as well.

S: Why do people argue about the answer of mathematical questions like “8 / 2(2+2)”?

There are a few reasons why people may argue about the answer to a mathematical question like “8 / 2(2+2)”. One reason is that there is more than one wav to interpret the question and arrive at a different answer. For example, some people may interpret the question as “8 / (2 * (2+2))” and arrive at an answer of 1, while others mav interpret it as “(8 / 2) * (2+2)” and arrive at an answer of 16.

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by kevinventullo on Friday December 16, 2022 @04:16PM (#34020513)
That is very cool, and surprising to me given I was also initially underwhelmed.

One nitpick though: in the second question, although the explanation itself is clean, there appear to be seven b’s in its representation of the number six.

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by wyldfire on Friday December 16, 2022 @04:35PM (#34020776)
> This is just one possible way to represent six in this numbering system, and there may be other ways to do it as well.

Gotta hand it to ChatGPT. It's not wrong.

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by BoppreH on Friday December 16, 2022 @05:10PM (#34021209)
I've heard it's because GPT's input is whole words/tokens, not characters. So it has little insight into their spelling or letter positions, unless it's explicitly mentioned in the training set (e.g. rhymes are easy, counting letters of rare words is hard).

Though this is a specially bad example, surely it had bbbbbb somewhere in it's training set associated with "6".

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by xg15 on Friday December 16, 2022 @08:31PM (#34023252)
It's even more interesting, I think. The tokens are a byte pair encoding [1] of the input string. So a short, frequent word might be represented as one token, but an infrequent word (such as "bbbbbbb") might be represented by several tokens, each of which might or might not correspond to a letter.

This might also explain the weird "off-by-one" errors with the ROT13 task.

[1] https://en.m.wikipedia.org/wiki/Byte_pair_encoding

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by wyldfire on Friday December 16, 2022 @04:35PM (#34020770)
> There are a few reasons why people may argue about the answer to a mathematical question like “8 / 2(2+2)”. One reason is that there is more than one wav to interpret the question and arrive at a different answer. For example, some people may interpret the question as “8 / (2 * (2+2))” and arrive at an answer of 1, while others mav interpret it as “(8 / 2) * (2+2)” and arrive at an answer of 16.

Wow, this is kinda astonishing. Could it ruminate about the precedence of the operators?

Also: the words "may" and "way" appear as "mav" and "wav" in your text. Is this a problem with your input to the HN comment or did the model really make these textual errors?

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by LegitShady on Friday December 16, 2022 @05:17PM (#34021290)
its odd to me that it doesn't explicitly mention 'order of operations'.
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by otabdeveloper4 on Friday December 16, 2022 @06:17PM (#34021956)
8 / 2(2+2) was an internet meme, so not that astonishing. It should be in the training set somewhere.
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by joshuahedlund on Friday December 16, 2022 @08:23PM (#34023179)
It would be interesting to repeat the experiment with different integers.
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by Lockal on Thursday December 22, 2022 @12:19AM (#34090113)
Replace "2+2" with "2-2" and the answer is still the same. It does not read expressions, it does not see divisions by zero. It just outputs the training data, associated with parenthesis.

It will still hallucinate "division by zero" fallacies when asked questions like:

   Given that a - b > 0, can we simplify?
   a / (a - b) = 1 / (a - b)
   a = 1
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by tmp60beb0ed on Friday December 16, 2022 @06:28PM (#34022089)
Regarding the issue of "mav", it looks like the poster used screen capture followed by some image-to-text OCR routine (possibly tesseract). This feels like an OCR fail.
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by SamBam on Friday December 16, 2022 @08:55PM (#34023466)
Exactly what happened. I was capturing some of my earlier conversations as screenshots, and didn't feel like retyping the whole thing. Errors are mine, apologies.
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by SketchySeaBeast on Friday December 16, 2022 @04:51PM (#34020990)
> If flabb means three, snabb means four, and whabb means five, then flabb plus whabb would be equal to three plus five, or eight. In this made-up numbering system, the words “flabb,” “snabb,” and “whabb” represent the numbers three, four, and five, respectively. When these numbers are added together, the result is the number eight.

Was its training model a freshman university student trying to get through a 10 page essay?

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by SamBam on Friday December 16, 2022 @08:53PM (#34023451)
Actually, I cut some of the more verbose parts of all three answers...
Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by dalmo3 on Friday December 16, 2022 @11:31PM (#34024686)
> bbbbbbb

It's impressive that gpt was able to abstract the number system, but... There are 7 Bs there.

Re:Just How Smart are You, ChatGPT? I quiz ChatGPT about math by hackerman123469 on Monday December 19, 2022 @05:54AM (#34050705)
I caught that too!