Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by petulla on Tuesday May 14, 2024 @04:51PM
(#40360008)
You need to know that this test set data wasn't included in the training data for this to be meaningful.
[ Parent
| on HN ]
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by sftombu on Tuesday May 14, 2024 @04:55PM
(#40360051)
If you ask the questions without providing the limerick first, it never gets the right answer. When the LLM gets the wrong answer, it is usually because it reverts to its training data and gives a generic answer that doesn't apply to the limerick.
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by trifurcate on Tuesday May 14, 2024 @10:24PM
(#40362450)
Why are you ruling out the possibility that training on the material may confer an advantage when the data is presented, even if the advantage may not be strong enough to pass the test without the data present in the context window?
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by a_wild_dandan on Tuesday May 14, 2024 @06:01PM
(#40360658)
No you don't. Compare the model's performance before and after uploading the material.
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by sftombu on Tuesday May 14, 2024 @07:43PM
(#40361432)
Previous answer to this question:
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by sumedh on Wednesday May 15, 2024 @08:37AM
(#40366104)
No such item.
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by lmeyerov on Tuesday May 14, 2024 @05:44PM
(#40360505)
I thought the test limericks were autogenerated?
Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack)
by sftombu on Tuesday May 14, 2024 @06:02PM
(#40360660)
They come from a database of 98k limericks -- https://zenodo.org/records/5722527