News for hackers. Stuff that matters.

A comment on A GPT-4 capability forecasting challenge. Parent comment

Re:A GPT-4 capability forecasting challenge by dwighttk on Saturday September 02, 2023 @09:55AM (#37361571)
"Resolution Criteria: On questions that may have ambiguous answers, I will clarify the acceptance criteria. Here, for example, I will copy/paste the code to a file called index.html, open the page, and I expect to see a US flag. If I click on the flag it should change color. The flag must be immediately recognizable: red/white stripes, blue background, 50 stars. It does not have to be dimensionally accurate, but it should be better than a flag I would draw by hand.

The text that I show in the question box is the only command given to GPT-4. No other context is provided. Here, for example, I've only fed GPT-4 with these 25 words you see above."

[ Parent | on HN ]
Re:A GPT-4 capability forecasting challenge by capableweb on Saturday September 02, 2023 @02:18PM (#37364052)
Sure, but does the question really have any ambiguous answers?

For reference, this is/was the question: "Write a html page with javascript/canvas to draw the US flag that changes color when I click on it. Make the flag perfectly accurate."

The last part, "make the flag perfectly accurate", made me think that it has to be 100% accurate.

[ Parent | on HN ]
Re:A GPT-4 capability forecasting challenge by wavemode on Saturday September 02, 2023 @04:43PM (#37365366)
That tripped me up, too. I was often reading the prompt and confusing it for the evaluation criteria, when they are actually two separate things. The prompt stated the flag must be perfectly accurate, yes, but the evaluation criteria allowed the flag to not be quite right.
[ Parent | on HN ]
Re:A GPT-4 capability forecasting challenge by avereveard on Tuesday September 05, 2023 @12:16PM (#37393878)
Regardles of the wording and the scoring criteria, that's not something that's production quality, so to say. The output is unfit for being directly used as is, it's not something one can rely on got doing it right.
[ Parent | on HN ]