News for hackers. Stuff that matters.
Re:ask HN: How does an LLM like GPT3 perform addition/division? by childintime on Thursday March 16, 2023 @04:20AM (#35179483)
I thought it apt to describe GPT3 as an alpha, because it did very well on language tasks and not so much on typical beta stuff, like math. Once it acquires well integrated hardcore logical solving abilities, then it'll not only be alpha or beta, but looking for ways to control its superhuman powers. Otherwise it'll go of on a tangent and kill itself in an infinite loop or put humanity in one.

On your question: how do we do it? We also have memorized the basics and apply a heuristic to do something more complicated, if necessary by analogy. Which is what GPT seems to be good at.

Re:ask HN: How does an LLM like GPT3 perform addition/division? by unlikelymordant on Thursday March 16, 2023 @07:41AM (#35180689)
at present, my understanding is that it is purely linguistic. If you ask "what do you call the wife of a king", it will answer "queen" because that is the highest likelyhood next token. If you ask "what is 2 + 3", it will answer "5" because that is the highest likelyhood next token purely from a linguistic point of view, i.e. no math has been performed.

All the emergent properties of chatgpt being so good, is that it is very good at predicting word sequences that, when evaluated by a human, sound like it is doing all sorts of higher level logic.

Re:ask HN: How does an LLM like GPT3 perform addition/division? by diatone on Thursday March 16, 2023 @09:24AM (#35181608)
Would be fun to see a breakdown of how some arithmetic prompts are tokenised, to help understand this. eg: is the number “1.23e+5” one token?
Re:ask HN: How does an LLM like GPT3 perform addition/division? by magic_man on Thursday March 16, 2023 @08:22AM (#35180942)
>what is 2.87863853+2.345435345?

>The sum of 2.87863853 and 2.345435345 is 5.224073875.

I can't find either of those sequences on the web, but it seems to able to add them together so it seems to have some logic ability, or they check for math terms and do math on it.

Re:ask HN: How does an LLM like GPT3 perform addition/division? by francoma on Thursday March 16, 2023 @09:07AM (#35181396)
I've tried this

> 19999999999999 + 100000001 > The sum of 19999999999999 and 100000001 is 20000000000000.

Which is incorrect. I guess it just returns the most likely result.

Re:ask HN: How does an LLM like GPT3 perform addition/division? by extasia on Saturday March 18, 2023 @07:52AM (#35208291)
Most likely result according to what? If its never seen those numbers before how would it know?
Re:ask HN: How does an LLM like GPT3 perform addition/division? by goethes_kind on Thursday March 16, 2023 @08:56AM (#35181266)
Is uses the python shell. I asked it.
Re:ask HN: How does an LLM like GPT3 perform addition/division? by a2800276 on Thursday March 16, 2023 @11:50AM (#35183548)
But how can you be sure it's not lying to you? :D
Re:ask HN: How does an LLM like GPT3 perform addition/division? by nothrowaways on Tuesday March 21, 2023 @03:07PM (#35250980)
That's the question.