Re:Capabilities of GPT-4 on Medical Challenge Problems
by mgfist on Sunday March 26, 2023 @07:58PM
(#35321089)
I'm not the person you responded to, but that's an old paper. LLMs have come a long way since. Nothing conclusive but GPT4 does show some signs of something deeper. https://arxiv.org/pdf/2303.12712.pdf
[ Parent
| on HN ]
Re:Capabilities of GPT-4 on Medical Challenge Problems
by cscurmudgeon on Sunday March 26, 2023 @08:02PM
(#35321126)
Yes, I have read that paper and have forwarded it eagerly to colleagues. Respectfully, please read the paper you have linked, especially section 8.2.
Look at section 8.2 in the paper you linked:
"8.2 Lack of planning in arithmetic/reasoning problems"
> However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution
Nowhere in the paper is an evaluation on standard reasoning datasets (https://tptp.org) because they are much much much tougher than the simple examples that GPT-4 struggles with.
LLMs are a huge breakthrough but let us not muddy the science.
Re:Capabilities of GPT-4 on Medical Challenge Problems
by Sunhold on Sunday March 26, 2023 @08:35PM
(#35321388)
GPT 4 was evaluated on multiple reasoning benchmarks in the release paper.[1] The problems in TPTP are not really appropriate for a language model and exceed the capabilities of most humans working without tools. Clearly humans have some reasoning ability even if they cannot solve those problems.