News for hackers. Stuff that matters.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by raincole on Thursday November 09, 2023 @09:26AM (#38205376)
Me: What's 6x6?

You: 36

Me: You cheated! You just cited the answer you memorized! You should have started from addition.

You: ...okay? 6+6=12, 12+6=18, 18+...

Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms.

You: ...you're being annoying, but okay? First axiom, we define 0 as...

Me: You cheated again! You memorized Peano Axioms! Jesus Christ, is there any intelligent creature left?

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by boxed on Thursday November 09, 2023 @09:48AM (#38205666)
But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by furyofantares on Thursday November 09, 2023 @10:25AM (#38206164)
It said they gave the exercise name, which doesn't sound like just the exercise number but probably mildly descriptive -- and they also gave it function stubs.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by raincole on Thursday November 09, 2023 @09:55AM (#38205745)
Me: I was being in such a blah blah situation... does the article 3 of Digital Government Act applies here?

My lawyer: Hmm the article 3 says--

Me: I knew it! Lawyers are not intelligent...

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by stnmtn on Thursday November 09, 2023 @10:45AM (#38206460)
If I gave you a programming problem and all I told you was that the problem name was Traveling Salesman, you might be able to solve it based on that.

If not that, then if I just said "fizzbuzz" to you, I'm sure you would be able to give the solution without me needing to say any other descriptions of the problem

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by boxed on Thursday November 09, 2023 @02:44PM (#38210059)
Again, because of memorization, not being able to code.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by FeepingCreature on Thursday November 09, 2023 @09:50PM (#38214512)
But in that case, not memorization of the specific problem set, but "programming background knowledge." Hardly something to blame the machine for when we rely on it every day.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by qup on Thursday November 09, 2023 @10:50AM (#38206532)
I know this is deep down a bad comment thread, but I thought I'd chime in here.

I have been writing function names and test names, and then telling gpt to fill in the test, which is usually does how I want (maybe with errors, but it tests the correct thing), and then I tell it to fill out the answers.

this is in a thing I'm building that's never been built, with names that I made up (but describe the functionality well)

It cannot have this spot memorized, I just invented it myself

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by jsight on Thursday November 09, 2023 @11:00AM (#38206658)
TBH, people underestimate how much of coding is just memorization. I'm guessing those of us with bad memories understand this more than the ones with good memories. :)

I can't remember how many times I've googled, "how do I create a directory in Python?". Now bard often generates an inline answer for me.