Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by boxed on Thursday November 09, 2023 @09:48AM
(#38205666)
But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.
[ Parent
| on HN ]
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by furyofantares on Thursday November 09, 2023 @10:25AM
(#38206164)
It said they gave the exercise name, which doesn't sound like just the exercise number but probably mildly descriptive -- and they also gave it function stubs.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by raincole on Thursday November 09, 2023 @09:55AM
(#38205745)
Me: I was being in such a blah blah situation... does the article 3 of Digital Government Act applies here?
My lawyer: Hmm the article 3 says--
Me: I knew it! Lawyers are not intelligent...
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by stnmtn on Thursday November 09, 2023 @10:45AM
(#38206460)
If I gave you a programming problem and all I told you was that the problem name was Traveling Salesman, you might be able to solve it based on that.
If not that, then if I just said "fizzbuzz" to you, I'm sure you would be able to give the solution without me needing to say any other descriptions of the problem
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by boxed on Thursday November 09, 2023 @02:44PM
(#38210059)
Again, because of memorization, not being able to code.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by FeepingCreature on Thursday November 09, 2023 @09:50PM
(#38214512)
But in that case, not memorization of the specific problem set, but "programming background knowledge." Hardly something to blame the machine for when we rely on it every day.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale
by qup on Thursday November 09, 2023 @10:50AM
(#38206532)
I know this is deep down a bad comment thread, but I thought I'd chime in here.
I have been writing function names and test names, and then telling gpt to fill in the test, which is usually does how I want (maybe with errors, but it tests the correct thing), and then I tell it to fill out the answers.
this is in a thing I'm building that's never been built, with names that I made up (but describe the functionality well)
It cannot have this spot memorized, I just invented it myself