News for hackers. Stuff that matters.
Re:Goodbye to Google Code Jam by TacticalCoder on Friday March 24, 2023 @11:24PM (#35298981)
> I wonder how gpt-4 does.

The problem is that as soon as people started tricking ChatGPT 3 into problems like that, the correct answers are now being used to train the next versions and are going to be part of the dataset.

So GPT-4 or GPT-5 may get the answer right, but that still wouldn't mean anything.

Re:Goodbye to Google Code Jam by fastball on Saturday March 25, 2023 @03:08AM (#35300249)
Fairly certain that is not what is going on here. GPT-4 seems genuinely better at reasoning and harder to trick from my testing.
Re:Goodbye to Google Code Jam by M4v3R on Saturday March 25, 2023 @04:31AM (#35300649)
Not the case for GPT-4 though, as it’s knowledge cutoff is the same as GPT-3s, that’s why it’s easy to compare the two on the same problems and see what’s the difference.
Re:Goodbye to Google Code Jam by versteegen on Saturday March 25, 2023 @04:56AM (#35300797)
Not so fast!

Yes, the GPT-4 paper says

> GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its pre-training data cuts off in September 2021 [footnote: The pre-training and post-training data contain a small amount of more recent data], and does not learn from its experience.

But note the more recent data. We know that InstructGPT (GPT-3.5) was RL trained on examples of previous queries to GPT3, such as those trick questions. We could assume everything (after filtering e.g. for benchmark contamination) ever sent to OpenAI is in that post-training set. This is indeed a very small amount of data compared to the trillion-plus tokens of older data it was surely trained on. We also know that when ARC did their evaluations of GPT-4, OpenAI hadn't finished fine-tuning yet, so they've certainly been continuing to do so recently.

See also my other comment https://news.ycombinator.com/item?id=35300668

Re:Goodbye to Google Code Jam by ResearchCode on Saturday March 25, 2023 @09:35AM (#35302482)
I suspect you're right. Part of the supervised "learning" is hard-coding answers to gotchas posted on Twitter.