News for hackers. Stuff that matters.
Re:Performance of Large Language Models on a Neurology Board... by hackerlight on Wednesday December 27, 2023 @12:43AM (#38779436)
> 85.0% of questions correctly compared with the mean human score of 73.8%

> These questions are either behind a paywall (in the case of the question bank) or published after 2021 and therefore out-of-training data for both LLMs.

> Despite their strengths, both models demonstrated weaker performance in tasks requiring higher-order thinking compared with questions requiring only lower-order thinking.

> Interestingly, both models exhibited confident language when answering questions, even when their responses were incorrect.

Re:Performance of Large Language Models on a Neurology Board... by cjbprime on Wednesday December 27, 2023 @02:25AM (#38779841)
> These questions are either behind a paywall (in the case of the question bank) or published after 2021 and therefore out-of-training data for both LLMs

I'm not sure that it's true that GPT-4's web scrape is currently from before 2021. And even if it was, this still doesn't seem like an effective analysis of test data contamination. (For example, have any questions that differ only in insignificant ways been asked before?)

I'm not skeptical of GPT-4's capability in the way many (most?) people are, but I'd still like to see the level of dialogue around training set data improve.

Re:Performance of Large Language Models on a Neurology Board... by nopinsight on Wednesday December 27, 2023 @02:39AM (#38779891)
I'm intrigued as well. Here's my Q&A with GPT-4 itself:

"Q: What was the data cutoff date when GPT-4 was first released?

A: The data cutoff for GPT-4, which means the point at which it stopped being trained on new data, was September 2021. This cutoff date is significant as it means GPT-4's training did not include information or events that occurred after this time."

Re:Performance of Large Language Models on a Neurology Board... by hooo on Wednesday December 27, 2023 @03:43AM (#38780191)
I just asked “what is your training cut off date?” and it responded “ My training includes data up to April 2023.”
Re:Performance of Large Language Models on a Neurology Board... by tigershark on Wednesday December 27, 2023 @05:27AM (#38780694)
April 2023 is for gtp-4 turbo (aka 1106-preview)
Re:Performance of Large Language Models on a Neurology Board... by cjbprime on Wednesday December 27, 2023 @01:44PM (#38785073)
(I don't think there's any reason to expect GPT-4 to actually know its current cutoff date.)
Re:Performance of Large Language Models on a Neurology Board... by viraptor on Wednesday December 27, 2023 @08:22AM (#38781655)
> These questions are either behind a paywall

That's not a great criterium. Exam questions are going to be discussed in many places, similar to how you can get answers to interview questions by learning from Reddit.