Are we in an AI Overhang?

Re:Are we in an AI Overhang? by Nasrudith on Monday July 27, 2020 @01:35PM (#23967379)

I wonder how well it would perform in accuracy if given a large number of simple but lengthy sums like 13453 + 53521. Increased set size would move it beyond simple input/output memorization. Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.

[ Parent | on HN ]

Re:Are we in an AI Overhang? by dwohnitmok on Monday July 27, 2020 @01:44PM (#23967492)

> Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.

I suspect this is how many humans do arithmetic (especially considering how many people conflate numbers with their representation as digits). So if GPT-3 is doing that, that's pretty impressive.

[ Parent | on HN ]

Re:Are we in an AI Overhang? by ja3k on Monday July 27, 2020 @02:38PM (#23968029)

You don't have to wonder. In their paper: https://arxiv.org/abs/2005.14165 they state it has 0.7% accuracy on zero shot 5 digit addition problems and 9.3% accuracy on few shot 5 digit addition problems.

[ Parent | on HN ]

Re:Are we in an AI Overhang? by ralfd on Monday July 27, 2020 @03:53PM (#23968745)

By the way: Arithmetic accuracy is better if dollar sign and commas are added (financial data in the training set):

http://gptprompts.wikidot.com/logic:math

[ Parent | on HN ]

Re:Are we in an AI Overhang? by gwern on Monday July 27, 2020 @03:45PM (#23968679)

You do have to wonder, because as that section states, the BPEs may impede arithmetic, and as we've found using the API, if you use commas, the accuracy (zero and few-shot) goes way up.

[ Parent | on HN ]