Re:Transformers Can Do Arithmetic with the Right Embeddings
by CamperBob2 on Wednesday May 29, 2024 @01:21AM
(#40508741)
Can you point me towards a citation for the 25% figure? I'm seeing numbers like 96% ( https://paperswithcode.com/sota/common-sense-reasoning-on-ar... ) but I'm guessing that's just for a subset of the larger class of questions.
Also, are you familiar with this study? What are your thoughts on it? https://www.esmo.org/newsroom/press-and-media-hub/esmo-media... Seems like a valid case where AI is competitive with skilled humans at object/image recognition.
Re:Transformers Can Do Arithmetic with the Right Embeddings
by andoando on Wednesday May 29, 2024 @01:17PM
(#40514411)
It seems theres multiple things by the name ARC. There is one by AI2 which is a text based science questions/word problems. The one Im referring to is this https://lab42.global/arc/
https://lab42.global/arcathon/leaderboard/
https://openreview.net/forum?id=E8m8oySvPJ
As to the study, I have the same objection as the radiology one. This isnt about object recognition and certainly not spatial reasoning, its the ability to predict cancer based on presence of visual features.
The "object recognition" part of this is super simple. Its a single, mostly 2D object in more or less the same angle, and the AI is trained on detecting just this.
Re:Transformers Can Do Arithmetic with the Right Embeddings
by CamperBob2 on Wednesday May 29, 2024 @02:21PM
(#40515063)
The "object recognition" part of this is super simple. Its a single, mostly 2D object in more or less the same angle, and the AI is trained on detecting just this.
And yet it outperforms human dermatologists.
Lastly, check out the ARC challenge or any other spatial reasoning tests for AI. Humans get ~80% on these challenges whereas the best AI is still at 25%