News for hackers. Stuff that matters.
Re:Capabilities of GPT-4 on Medical Challenge Problems by hackerlight on Sunday March 26, 2023 @05:39PM (#35319918)
> "Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. "
Re:Capabilities of GPT-4 on Medical Challenge Problems by og_kalu on Sunday March 26, 2023 @05:46PM (#35319966)
If you read the technic paper of gpt-4, the confidence of the base model directly correlated with its ability to solve problems. Sadly, the hammer of alignment knocked it right out.
Re:Capabilities of GPT-4 on Medical Challenge Problems by sebzim4500 on Sunday March 26, 2023 @06:15PM (#35320215)
Is there any indication of whether the correlation was destroyed during the supervised finetuning or during the RLHF phase? Or are there even two phases any more?
Re:Capabilities of GPT-4 on Medical Challenge Problems by og_kalu on Sunday March 26, 2023 @06:18PM (#35320246)
Seems it was fine with instruction fine-tuning. Then gone with the RLHF.
Re:Capabilities of GPT-4 on Medical Challenge Problems by cubefox on Sunday March 26, 2023 @06:34PM (#35320401)
I don't think the paper says that. I would guess both SL and RL cause mode collapse.
Re:Capabilities of GPT-4 on Medical Challenge Problems by WalterBright on Sunday March 26, 2023 @06:04PM (#35320119)
Hammer of alignment??
Re:Capabilities of GPT-4 on Medical Challenge Problems by james-revisoai on Sunday March 26, 2023 @06:11PM (#35320178)
Recent models (since the Instruct series and including ChatGPT) are essentially two parts - a "Base" part (what you would have read about in 2020 when GPT-3 released), and a "RLHF" part - which improves the output from the "Base" part by slightly changing how it gets produced.

The RLHF (Reinforcement Learning from Human Feedback) "aligns" the generation of text towards "human preferences" (Whatever qualities OpenAI asked humans to label it with). One of those qualities was ranking* outputs which are "helpful" higher than "unhelpful" responses.

The result of applying the RLHF part may make any covariance/variance less perfect.

Re:Capabilities of GPT-4 on Medical Challenge Problems by og_kalu on Sunday March 26, 2023 @06:20PM (#35320261)
Kind of. Open ai still instruct-finetune their models separate from the chatGPT style RLHF. The instruct tuning itself seems to only improve the raw model.
Re:Capabilities of GPT-4 on Medical Challenge Problems by james-revisoai on Monday March 27, 2023 @03:14PM (#35331026)
Yeah good detail that they are instruct-finetuned too (say text-davinci-003) ... what I mean here is the two situations below in Figure 8 for the "Base model" and "RHLF model" (a difference I appreciate I did not highlight originally) as in Figure 8 of the paper.

From reddit passing on a LessWrong comment: Paul Christiano, alignment researcher at ARC/ previously OpenAI was questioned by somebody: Perhaps I am misunderstanding Figure 8? I was assuming that they asked the model for the answer, then asked the model what probability it thinks that that answer is correct. Under this assumption, it looks like the pre-trained model outputs the correct probability, but the RLHF model gives exaggerated probabilities because it thinks that will trick you into giving it higher reward.

And Paul replies:

Yes, I think you are misunderstanding figure 8. I don't have inside information, but without explanation "calibration" would almost always mean reading it off from the logits. If you instead ask the model to express its uncertainty I think it will do a much worse job, and the RLHF model will probably perform similarly to the pre-trained model. (This depends on details of the human feedback, under a careful training regime it would probably get modestly better.)

Re:Capabilities of GPT-4 on Medical Challenge Problems by tangjurine on Sunday March 26, 2023 @06:07PM (#35320143)
gpt is trained initially on predicting text, then trained with rlhf to make the model more helpful, etc. The second step is referred to as alignment.

Rlhf: https://huggingface.co/blog/rlhf

Re:Capabilities of GPT-4 on Medical Challenge Problems by dataveg on Sunday March 26, 2023 @06:07PM (#35320142)
It’s a similar concept to the Squirrel of Despair.
Re:Capabilities of GPT-4 on Medical Challenge Problems by owlboy on Sunday March 26, 2023 @06:18PM (#35320245)
It's 10 mins since you posted this. And now this comment is the top result on Google search.