News for hackers. Stuff that matters.

A comment on GPTs Are GPTs: An Early Look at the Labor Market Impact Potential of LLMs. Parent comment

Re:GPTs Are GPTs: An Early Look at the Labor Market Impact P... by tap-snap-or-nap on Monday March 20, 2023 @06:21AM (#35229528)
I asked it "What is the accuracy and confidence level (you must rate it out of 10) that the above answer is correct ?"

It always generates a random number when it is caught making up bullshit sometimes claims that the information 10/10 accurate and LLM's always provides accurate information.

[ Parent | on HN ]
Re:GPTs Are GPTs: An Early Look at the Labor Market Impact P... by sorz on Monday March 20, 2023 @12:52PM (#35234532)
OpenAI did exact the test for GPT-4. The raw, non-fine-tune GPT-4 is quite good at predicting confidence level ("highly calibrated" by their words). But the RLHF fine-tuning process seems ruin its calibration. Figure 8 on page 12 of GPT-4 Technical Report shows this dramatic changes before & after fine-tuning.
[ Parent | on HN ]