Re:A GPT-4 capability forecasting challenge
by wavemode on Saturday September 02, 2023 @04:43PM
(#37365366)
That tripped me up, too. I was often reading the prompt and confusing it for the evaluation criteria, when they are actually two separate things. The prompt stated the flag must be perfectly accurate, yes, but the evaluation criteria allowed the flag to not be quite right.
Re:A GPT-4 capability forecasting challenge
by avereveard on Tuesday September 05, 2023 @12:16PM
(#37393878)
Regardles of the wording and the scoring criteria, that's not something that's production quality, so to say. The output is unfit for being directly used as is, it's not something one can rely on got doing it right.
For reference, this is/was the question: "Write a html page with javascript/canvas to draw the US flag that changes color when I click on it. Make the flag perfectly accurate."
The last part, "make the flag perfectly accurate", made me think that it has to be 100% accurate.